[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:17 dist-test-slave-g0w1 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:17 dist-test-slave-g0w1 krb5kdc[395](info): set up 2 sockets
Sep 27 15:06:17 dist-test-slave-g0w1 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 15:06:18.676191 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:18 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:18.862790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39145
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:19.216624 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:19.216939 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:19.217106 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:19.225116 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:19.225243 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:19.225344 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:19.225450 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:19.225531 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:19.236038 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:19.237529 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:19.238994 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:19.252472 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.252480 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.252676 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:19.253098 408 server_base.cc:1034] running on GCE node
I20260927 15:06:19.254215 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:19.256036 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:19.257323 408 hybrid_clock.cc:648] HybridClock initialized: now 1790521579257254 us; error 88 us; skew 500 ppm
I20260927 15:06:19.257779 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:19 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:19.262039 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:19.264039 408 webserver.cc:460] Webserver started at http://127.0.97.62:45831/ using document root <none> and password file <none>
I20260927 15:06:19.264885 408 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:19.265054 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:19.265450 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:19.269054 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "c932d5d5c5744884a2de96e7ced44771"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-g0w1"
server_key: "aa1f8c3e93141f2a5dcdcd50b4814d34"
server_key_iv: "b9f0bd1cafdce94b2737e4ffa8835ad0"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.270080 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "c932d5d5c5744884a2de96e7ced44771"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-g0w1"
server_key: "aa1f8c3e93141f2a5dcdcd50b4814d34"
server_key_iv: "b9f0bd1cafdce94b2737e4ffa8835ad0"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.276475 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:06:19.280757 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.282486 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:06:19.282727 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "c932d5d5c5744884a2de96e7ced44771"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-g0w1"
server_key: "aa1f8c3e93141f2a5dcdcd50b4814d34"
server_key_iv: "b9f0bd1cafdce94b2737e4ffa8835ad0"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.282968 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:19.321632 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:19.326673 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:19.327011 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:19.348865 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39145
I20260927 15:06:19.348886 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39145 every 8 connection(s)
I20260927 15:06:19.350478 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:19.352860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 408
I20260927 15:06:19.353214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:19.355830 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:19.355875 388 external_mini_cluster.cc:1428] Setting key 8035a614b93e350077e7e77a9eab671e
I20260927 15:06:19.368950 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771: Bootstrap starting.
I20260927 15:06:19.372462 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:19.374049 475 log.cc:824] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:19.378288 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771: No bootstrap required, opened a new log
I20260927 15:06:19.389179 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c932d5d5c5744884a2de96e7ced44771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39145 } }
I20260927 15:06:19.390065 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:19.390203 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c932d5d5c5744884a2de96e7ced44771, State: Initialized, Role: FOLLOWER
Sep 27 15:06:19 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:19.391173 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c932d5d5c5744884a2de96e7ced44771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39145 } }
I20260927 15:06:19.391541 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:19.391692 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:19.391898 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:19.395833 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c932d5d5c5744884a2de96e7ced44771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39145 } }
I20260927 15:06:19.396585 475 leader_election.cc:302] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c932d5d5c5744884a2de96e7ced44771; no voters:
I20260927 15:06:19.397348 475 leader_election.cc:288] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:19.397553 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:19.399237 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [term 1 LEADER]: Becoming Leader. State: Replica: c932d5d5c5744884a2de96e7ced44771, State: Running, Role: LEADER
I20260927 15:06:19.399907 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c932d5d5c5744884a2de96e7ced44771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39145 } }
I20260927 15:06:19.400286 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:19.404003 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c932d5d5c5744884a2de96e7ced44771. Latest consensus state: current_term: 1 leader_uuid: "c932d5d5c5744884a2de96e7ced44771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c932d5d5c5744884a2de96e7ced44771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39145 } } }
I20260927 15:06:19.404779 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:19.404325 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c932d5d5c5744884a2de96e7ced44771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c932d5d5c5744884a2de96e7ced44771" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39145 } } }
I20260927 15:06:19.405462 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:19.405941 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:19.407307 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.361062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.97.62:39145)
0927 15:06:19.362136 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.362161 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.363653 (+ 1492us) server_negotiation.cc:407] Connection header received
0927 15:06:19.374506 (+ 10853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.374545 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.375799 (+ 1254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.376367 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:19.379601 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.382086 (+ 2485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.384241 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.385578 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.385838 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.392976 (+ 7138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:19.392990 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:19.393016 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:19.393080 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:19.397879 (+ 4799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:19.399325 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:19.399336 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:19.399344 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:19.399611 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:19.400324 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:19.400329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:19.400332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:19.401213 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:19.402804 (+ 1591us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.403237 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.403946 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":470,"threads_started":1}
I20260927 15:06:19.423478 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:19.434146 487 catalog_manager.cc:1347] Generated new cluster ID: 3aed6048d4fc4dd58ac411b36dcd303a
I20260927 15:06:19.434298 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:19.460592 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:19.462787 487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:19.472316 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c932d5d5c5744884a2de96e7ced44771: Generated new TSK 0
I20260927 15:06:19.473253 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:19.519858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39145
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:19.887496 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:19.887805 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:19.888019 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:19.896057 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:19.896179 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:19.896404 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:19.907359 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:19.908918 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:19.910517 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:19.924870 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.924961 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.924908 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:19.926604 501 server_base.cc:1034] running on GCE node
I20260927 15:06:19.927409 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:19.928431 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:19.929644 501 hybrid_clock.cc:648] HybridClock initialized: now 1790521579929587 us; error 46 us; skew 500 ppm
I20260927 15:06:19.930013 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:19 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:19.933374 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:19.934863 501 webserver.cc:460] Webserver started at http://127.0.97.1:42421/ using document root <none> and password file <none>
I20260927 15:06:19.935672 501 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:19.935782 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:19.936043 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:19.939541 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "4dcd7a17b93a49b3a490cf95c118f822"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-g0w1"
server_key: "ac3321c234d16d27ec817b1e346e5451"
server_key_iv: "7c82c82ba789b480caa7d395c8ac7307"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.940363 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4dcd7a17b93a49b3a490cf95c118f822"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-g0w1"
server_key: "ac3321c234d16d27ec817b1e346e5451"
server_key_iv: "7c82c82ba789b480caa7d395c8ac7307"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.946265 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 15:06:19.950103 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.951964 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:06:19.952171 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "4dcd7a17b93a49b3a490cf95c118f822"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-g0w1"
server_key: "ac3321c234d16d27ec817b1e346e5451"
server_key_iv: "7c82c82ba789b480caa7d395c8ac7307"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.952428 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:19.974251 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:19.978993 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:19.979449 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:19.980458 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:19.981869 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:19.981942 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.982107 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:19.982195 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:20.020753 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40855
I20260927 15:06:20.020792 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40855 every 8 connection(s)
I20260927 15:06:20.022181 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:20.023747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 501
I20260927 15:06:20.023985 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:20.024576 388 external_mini_cluster.cc:1428] Setting key 86190be81efb470dc6ab51341e447e7b
Sep 27 15:06:20 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:06:20.048823 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:20.025057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39565 (local address 127.0.97.62:39145)
0927 15:06:20.025515 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:20.025520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:20.026702 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 15:06:20.029630 (+ 2928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:20.029635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:20.029717 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:20.029840 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:20.032430 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.033393 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.035433 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.035898 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.036067 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:20.041218 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:20.041228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:20.041234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:20.041286 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:20.044459 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:20.045636 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:20.045641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:20.045644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:20.045797 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:20.046512 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:20.046520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:20.046526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:20.046936 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:20.047138 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:20.047997 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 15:06:20.048307 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260927 15:06:20.049831 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:20.025661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39145 (local address 127.0.97.1:39565)
0927 15:06:20.026513 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:20.026569 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:20.029246 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:20.030176 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:20.030194 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:20.031182 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:20.032118 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:20.032151 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.033632 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.033644 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:20.035239 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:20.035255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.036187 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.036198 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:20.036517 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:20.037352 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:20.037380 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:20.040866 (+ 3486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:20.044680 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:20.044689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:20.044706 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:20.045283 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:20.045977 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:20.045986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:20.045993 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:20.046304 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:20.047140 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:20.047154 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:20.047549 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 15:06:20.047820 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 15:06:20.048293 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":475,"threads_started":1}
I20260927 15:06:20.051837 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39145
I20260927 15:06:20.052208 629 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:20.053186 629 heartbeater.cc:511] Master 127.0.97.62:39145 requested a full tablet report, sending...
I20260927 15:06:20.055811 440 ts_manager.cc:194] Registered new tserver with Master: 4dcd7a17b93a49b3a490cf95c118f822 (127.0.97.1:40855)
I20260927 15:06:20.057410 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:20.071135 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39145
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:20.433460 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:20.433765 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:20.433861 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:20.441883 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:20.442008 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:20.442126 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:20.452943 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:20.454265 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:20.455750 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:20.470252 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:20.470304 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:20.470332 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:20.471264 635 server_base.cc:1034] running on GCE node
I20260927 15:06:20.471956 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:20.473165 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:20.474351 635 hybrid_clock.cc:648] HybridClock initialized: now 1790521580474295 us; error 57 us; skew 500 ppm
I20260927 15:06:20.474704 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:20 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:20.478708 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:20.480424 635 webserver.cc:460] Webserver started at http://127.0.97.2:40969/ using document root <none> and password file <none>
I20260927 15:06:20.481201 635 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:20.481343 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:20.481698 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:20.484925 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f2063dc7efa41c784faeb13e4cc7811"
format_stamp: "Formatted at 2026-09-27 15:06:20 on dist-test-slave-g0w1"
server_key: "d6c92c29417eac1f97ee5b593e06bc24"
server_key_iv: "b20619ca70d1075974167e46c7c12ab1"
server_key_version: "encryptionkey@0"
I20260927 15:06:20.485893 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f2063dc7efa41c784faeb13e4cc7811"
format_stamp: "Formatted at 2026-09-27 15:06:20 on dist-test-slave-g0w1"
server_key: "d6c92c29417eac1f97ee5b593e06bc24"
server_key_iv: "b20619ca70d1075974167e46c7c12ab1"
server_key_version: "encryptionkey@0"
I20260927 15:06:20.492012 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:06:20.496258 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:20.497989 635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:06:20.498210 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "3f2063dc7efa41c784faeb13e4cc7811"
format_stamp: "Formatted at 2026-09-27 15:06:20 on dist-test-slave-g0w1"
server_key: "d6c92c29417eac1f97ee5b593e06bc24"
server_key_iv: "b20619ca70d1075974167e46c7c12ab1"
server_key_version: "encryptionkey@0"
I20260927 15:06:20.498492 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:20.544438 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:20.549419 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:20.549741 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:20.550863 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:20.552358 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:20.552434 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:20.552565 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:20.552625 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:20.590840 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42065
I20260927 15:06:20.590874 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42065 every 8 connection(s)
I20260927 15:06:20.592276 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:20.601269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 635
I20260927 15:06:20.601536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:20.601976 388 external_mini_cluster.cc:1428] Setting key fce306036b548635bdc47173142c960e
Sep 27 15:06:20 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:20.616364 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:20.595044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51569 (local address 127.0.97.62:39145)
0927 15:06:20.595884 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:20.595891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:20.597153 (+ 1262us) server_negotiation.cc:407] Connection header received
0927 15:06:20.598987 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:20.598992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:20.599118 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:20.599235 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:20.601067 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.601991 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.604215 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.604655 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.604800 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:20.608707 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:20.608712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:20.608716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:20.608753 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:20.611823 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:20.613004 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:20.613009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:20.613012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:20.613166 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:20.613789 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:20.613794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:20.613796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:20.614132 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:20.614272 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:20.615823 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0927 15:06:20.616034 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":536,"threads_started":1}
I20260927 15:06:20.618050 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:20.596168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39145 (local address 127.0.97.2:51569)
0927 15:06:20.596952 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:20.597000 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:20.598678 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:20.599540 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:20.599550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:20.600204 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:20.600854 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:20.600874 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.602246 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.602250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:20.604070 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:20.604086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.604903 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.604910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:20.605176 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:20.605914 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:20.605939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:20.608428 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:20.612041 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:20.612054 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:20.612078 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:20.612789 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:20.613311 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:20.613319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:20.613326 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:20.613614 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:20.614287 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:20.614299 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:20.614839 (+ 540us) client_negotiation.cc:769] Sending connection context
0927 15:06:20.615657 (+ 818us) client_negotiation.cc:240] Negotiation successful
0927 15:06:20.616355 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:06:20.620051 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39145
I20260927 15:06:20.620406 763 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:20.621320 763 heartbeater.cc:511] Master 127.0.97.62:39145 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:06:20.623225 440 ts_manager.cc:194] Registered new tserver with Master: 3f2063dc7efa41c784faeb13e4cc7811 (127.0.97.2:42065)
I20260927 15:06:20.624104 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:20.647261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39145
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:21.005638 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:21.005949 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:21.006048 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:21.014125 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:21.014252 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:21.014369 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:21.025306 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:21.026618 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:21.028074 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:21.042863 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:21.042881 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:21.042814 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.044359 769 server_base.cc:1034] running on GCE node
I20260927 15:06:21.045107 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:21.046207 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:21.047410 769 hybrid_clock.cc:648] HybridClock initialized: now 1790521581047342 us; error 69 us; skew 500 ppm
I20260927 15:06:21.047757 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:21.051335 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:21.053118 769 webserver.cc:460] Webserver started at http://127.0.97.3:43173/ using document root <none> and password file <none>
I20260927 15:06:21.054137 769 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:21.054294 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:21.054634 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:21.058189 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "0bfaed1393aa428bb41ff667dfdf6eb3"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-g0w1"
server_key: "1584384eb8976151b573616245799178"
server_key_iv: "3791b19f4b513e41bd11c6dfd517fc8a"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.059245 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0bfaed1393aa428bb41ff667dfdf6eb3"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-g0w1"
server_key: "1584384eb8976151b573616245799178"
server_key_iv: "3791b19f4b513e41bd11c6dfd517fc8a"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.061142 629 heartbeater.cc:503] Master 127.0.97.62:39145 was elected leader, sending a full tablet report...
I20260927 15:06:21.066104 769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 15:06:21.070108 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.071743 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:06:21.071939 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "0bfaed1393aa428bb41ff667dfdf6eb3"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-g0w1"
server_key: "1584384eb8976151b573616245799178"
server_key_iv: "3791b19f4b513e41bd11c6dfd517fc8a"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.072148 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:21.085502 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:21.090525 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:21.090854 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:21.091926 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:21.093344 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:21.093458 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.093611 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:21.093683 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.132123 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35557
I20260927 15:06:21.132169 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35557 every 8 connection(s)
I20260927 15:06:21.133612 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:21.136842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 769
I20260927 15:06:21.137042 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:21.137569 388 external_mini_cluster.cc:1428] Setting key 3fae126492bd4b7b9f594b486f53bb52
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:21.156796 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.136365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44887 (local address 127.0.97.62:39145)
0927 15:06:21.136878 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.136883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.138277 (+ 1394us) server_negotiation.cc:407] Connection header received
0927 15:06:21.140754 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.140758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.140826 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.140929 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.142856 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.143752 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.145401 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.145807 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.145918 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.149964 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.149969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.149973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.150008 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.153429 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.154439 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.154444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.154447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.154616 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.155029 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.155037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.155039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.155369 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.155537 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.156309 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.156531 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260927 15:06:21.157809 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.137188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39145 (local address 127.0.97.3:44887)
0927 15:06:21.138081 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.138129 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.140420 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.141241 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.141257 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.142027 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:21.142652 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.142670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.143914 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.143918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.145232 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.145245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.146031 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.146036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.146247 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.146955 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:21.146978 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:21.149731 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:21.153583 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.153592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.153630 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.154290 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.154735 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.154738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.154741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.154925 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.155562 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:21.155569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:21.155941 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.156219 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.156629 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":378,"threads_started":1}
I20260927 15:06:21.159826 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39145
I20260927 15:06:21.160133 897 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:21.160815 897 heartbeater.cc:511] Master 127.0.97.62:39145 requested a full tablet report, sending...
I20260927 15:06:21.162282 440 ts_manager.cc:194] Registered new tserver with Master: 0bfaed1393aa428bb41ff667dfdf6eb3 (127.0.97.3:35557)
I20260927 15:06:21.163225 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44887
I20260927 15:06:21.167397 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:21.218789 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.200245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.0.97.1:40855)
0927 15:06:21.201192 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.201200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.201230 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:06:21.201352 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.201359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.201589 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.201736 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.203843 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.205286 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.206725 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.207543 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.207656 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.213155 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.213171 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.213188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.213238 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.216211 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.216954 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.216962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.216964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.217101 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.217460 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.217464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.217466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.217807 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.218056 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.218289 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.218498 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":646,"threads_started":1}
W20260927 15:06:21.220181 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:21.266847 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.252454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.0.97.1:40855)
0927 15:06:21.252700 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.252707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.253001 (+ 294us) server_negotiation.cc:407] Connection header received
0927 15:06:21.253226 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.253232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.253314 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.253423 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.254763 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.255535 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.256765 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.257192 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.257338 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.261491 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.261497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.261500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.261541 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.264333 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.265107 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.265111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.265114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.265245 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.265685 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.265690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.265693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.265961 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.266135 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.266433 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.266601 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:06:21.268535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 501
I20260927 15:06:21.283025 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.269529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.0.97.1:40855)
0927 15:06:21.269722 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.269731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.270028 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:06:21.270279 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.270282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.270340 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.270446 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.271739 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.272448 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.273712 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.274183 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.274323 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.276934 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.276941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.276945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.276999 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.280307 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.281019 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.281028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.281031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.281174 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.281621 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.281625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.281628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.281898 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.282130 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.282441 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.282653 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:21.283794 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:21.373302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 501
I20260927 15:06:21.391188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 635
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:21.409010 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.392206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.0.97.2:42065)
0927 15:06:21.392822 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.392830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.392859 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:06:21.392949 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.392956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.393173 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.393312 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.394570 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.395762 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.397150 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.397777 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.397900 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.402394 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.402410 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.402427 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.402468 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.405722 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.406546 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.406555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.406557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.406706 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.407264 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.407268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.407270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.407701 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.407922 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.408330 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.408672 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":372,"threads_started":1}
I20260927 15:06:21.410473 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:21.495159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 635
I20260927 15:06:21.518373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 769
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:21.538038 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.519525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.97.3:35557)
0927 15:06:21.520077 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.520084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.520114 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:06:21.520173 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.520179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.520449 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.520586 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.521816 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.523058 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.524492 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.525142 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.525250 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.529731 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.529746 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.529762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.529809 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.534229 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.535406 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.535418 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.535422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.535595 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.536075 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.536092 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.536096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.536676 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.537012 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.537356 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.537672 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
I20260927 15:06:21.539605 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:21.629490 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 769
I20260927 15:06:21.646222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 408
I20260927 15:06:21.647733 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:21.753580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 408
2026-09-27T15:06:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4196 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[919](info): set up 2 sockets
Sep 27 15:06:21 dist-test-slave-g0w1 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260927 15:06:22.851641 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:22 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:23.045158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37901
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:23.377588 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:23.377974 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:23.378142 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:23.385905 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:23.386080 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:23.386194 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:23.386298 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:23.386387 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:23.396836 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:23.398356 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:23.399901 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:23.412652 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:23.412628 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:23.413589 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:23.414063 932 server_base.cc:1034] running on GCE node
I20260927 15:06:23.414940 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:23.416543 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:23.417793 932 hybrid_clock.cc:648] HybridClock initialized: now 1790521583417746 us; error 45 us; skew 500 ppm
I20260927 15:06:23.418164 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:23 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:23.428186 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:23.430141 932 webserver.cc:460] Webserver started at http://127.0.97.62:32901/ using document root <none> and password file <none>
I20260927 15:06:23.430951 932 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:23.431138 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:23.431489 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:23.434753 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "edcaecccf6d946ddb2043da9e0db6bf9"
format_stamp: "Formatted at 2026-09-27 15:06:23 on dist-test-slave-g0w1"
server_key: "8b525324cc5cc1a1c7a0b5605548a80b"
server_key_iv: "a6083fdfb504ca545d6fcb83a95eb491"
server_key_version: "encryptionkey@0"
I20260927 15:06:23.435600 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "edcaecccf6d946ddb2043da9e0db6bf9"
format_stamp: "Formatted at 2026-09-27 15:06:23 on dist-test-slave-g0w1"
server_key: "8b525324cc5cc1a1c7a0b5605548a80b"
server_key_iv: "a6083fdfb504ca545d6fcb83a95eb491"
server_key_version: "encryptionkey@0"
I20260927 15:06:23.441421 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:23.445482 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:23.447163 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:23.447410 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "edcaecccf6d946ddb2043da9e0db6bf9"
format_stamp: "Formatted at 2026-09-27 15:06:23 on dist-test-slave-g0w1"
server_key: "8b525324cc5cc1a1c7a0b5605548a80b"
server_key_iv: "a6083fdfb504ca545d6fcb83a95eb491"
server_key_version: "encryptionkey@0"
I20260927 15:06:23.447687 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:23.473628 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:23.478880 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:23.479218 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:23.499199 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37901
I20260927 15:06:23.499300 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37901 every 8 connection(s)
I20260927 15:06:23.500725 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:23.503897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 932
I20260927 15:06:23.504195 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:23.505028 388 external_mini_cluster.cc:1428] Setting key a178790ee676eb8bed8a9f4a7f628221
I20260927 15:06:23.505627 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:23.521863 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9: Bootstrap starting.
Sep 27 15:06:23 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:23.526113 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:23.527601 999 log.cc:824] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:23.531569 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9: No bootstrap required, opened a new log
I20260927 15:06:23.533461 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.507388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.97.62:37901)
0927 15:06:23.508210 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:23.508223 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:23.508268 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:06:23.511219 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:23.511248 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:23.511728 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:23.512103 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:23.513978 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.515449 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.517278 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.518138 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.518305 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:23.523318 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:23.523331 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:23.523354 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:23.523421 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:23.527796 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:23.528880 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:23.528893 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:23.528902 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:23.529211 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:23.529719 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:23.529725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:23.529728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:23.530552 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:23.530870 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:23.531382 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 15:06:23.531888 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":381,"threads_started":1}
I20260927 15:06:23.540552 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260927 15:06:23.540953 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:23.541061 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcaecccf6d946ddb2043da9e0db6bf9, State: Initialized, Role: FOLLOWER
I20260927 15:06:23.541700 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260927 15:06:23.541921 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:23.542085 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:23.542277 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:23.544821 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260927 15:06:23.545466 999 leader_election.cc:302] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edcaecccf6d946ddb2043da9e0db6bf9; no voters:
I20260927 15:06:23.546125 999 leader_election.cc:288] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:23.546244 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:23.548014 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [term 1 LEADER]: Becoming Leader. State: Replica: edcaecccf6d946ddb2043da9e0db6bf9, State: Running, Role: LEADER
I20260927 15:06:23.548666 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } }
I20260927 15:06:23.549009 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:23.551987 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } } }
I20260927 15:06:23.551995 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edcaecccf6d946ddb2043da9e0db6bf9. Latest consensus state: current_term: 1 leader_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcaecccf6d946ddb2043da9e0db6bf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37901 } } }
I20260927 15:06:23.552477 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:23.552477 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:23.553373 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:23.563796 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:23.573743 1013 catalog_manager.cc:1347] Generated new cluster ID: ae118ab8c8ff4cc0a6f01511b3fa9546
I20260927 15:06:23.573882 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:23.595153 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:23.596673 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:23.605571 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edcaecccf6d946ddb2043da9e0db6bf9: Generated new TSK 0
I20260927 15:06:23.606769 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:23.657076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:24.035081 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:24.035540 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:24.035727 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:24.048216 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:24.048396 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:24.048614 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:24.063764 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:24.065287 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:24.066790 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:24.081686 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:24.081651 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:24.081651 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:24.083024 1025 server_base.cc:1034] running on GCE node
I20260927 15:06:24.083804 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:24.084975 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:24.086215 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790521584086175 us; error 46 us; skew 500 ppm
I20260927 15:06:24.086510 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:24 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:24.089920 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:24.091606 1025 webserver.cc:460] Webserver started at http://127.0.97.1:43847/ using document root <none> and password file <none>
I20260927 15:06:24.092336 1025 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:24.092468 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:24.092795 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:24.095934 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "3cbc76da85374b25b09c0c3907ce9893"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-g0w1"
server_key: "77a3feef8852e6fc29517d480f99e15b"
server_key_iv: "186b262d67e606497426fa141090aae0"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.096796 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3cbc76da85374b25b09c0c3907ce9893"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-g0w1"
server_key: "77a3feef8852e6fc29517d480f99e15b"
server_key_iv: "186b262d67e606497426fa141090aae0"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.102402 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:24.106499 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.108098 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:24.108278 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "3cbc76da85374b25b09c0c3907ce9893"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-g0w1"
server_key: "77a3feef8852e6fc29517d480f99e15b"
server_key_iv: "186b262d67e606497426fa141090aae0"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.108484 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:24.122318 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:24.127177 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:24.127496 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:24.128611 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:24.130115 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:24.130199 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.130321 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:24.130388 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.165467 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35631
I20260927 15:06:24.165609 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35631 every 8 connection(s)
I20260927 15:06:24.167018 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:24.176874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1025
I20260927 15:06:24.177117 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:24.177870 388 external_mini_cluster.cc:1428] Setting key 5d89d4c5a278ccd6037b576225b3cb71
Sep 27 15:06:24 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:24.190896 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:24.169927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60023 (local address 127.0.97.62:37901)
0927 15:06:24.170385 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:24.170390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:24.171428 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 15:06:24.173364 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:24.173370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:24.173466 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:24.173587 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:24.175370 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.176112 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.178677 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.179337 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.179499 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:24.184273 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:24.184279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:24.184283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:24.184319 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:24.187061 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.188211 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.188215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.188217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.188373 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.188913 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.188916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.188918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.189186 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:24.189292 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:24.190376 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0927 15:06:24.190608 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260927 15:06:24.192648 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:24.170447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37901 (local address 127.0.97.1:60023)
0927 15:06:24.171254 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:24.171302 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:24.173054 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:24.173847 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:24.173856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:24.174475 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:24.175188 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:24.175209 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.176296 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.176301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:24.178501 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:24.178524 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.179582 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.179593 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:24.179890 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:24.180699 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:24.180735 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:24.183981 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:24.187307 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:24.187321 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:24.187346 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:24.188050 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:24.188499 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:24.188506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:24.188513 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:24.188794 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:24.189314 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:24.189326 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:24.189862 (+ 536us) client_negotiation.cc:769] Sending connection context
0927 15:06:24.190201 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 15:06:24.190850 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260927 15:06:24.194522 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37901
I20260927 15:06:24.194865 1153 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:24.195771 1153 heartbeater.cc:511] Master 127.0.97.62:37901 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:06:24.198307 964 ts_manager.cc:194] Registered new tserver with Master: 3cbc76da85374b25b09c0c3907ce9893 (127.0.97.1:35631)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:06:24.199930 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:24.222581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:24.567658 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:24.568030 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:24.568208 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:24.575776 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:24.575963 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:24.576169 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:24.586848 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:24.588312 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:24.589905 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:24.604125 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:24.604123 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:24.604171 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:24.604641 1159 server_base.cc:1034] running on GCE node
I20260927 15:06:24.605654 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:24.606786 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:24.608034 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790521584607940 us; error 80 us; skew 500 ppm
I20260927 15:06:24.608379 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:24 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:24.612020 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:24.613854 1159 webserver.cc:460] Webserver started at http://127.0.97.2:33457/ using document root <none> and password file <none>
I20260927 15:06:24.614576 1159 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:24.614681 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:24.614959 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:24.618125 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "7b5e9a6cdf774b0b8cf466162d81675b"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-g0w1"
server_key: "f1789afbc3ac012a16a8c1e58dbdfc1c"
server_key_iv: "546d7dc90b43f360d1f665b7124fd126"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.618947 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7b5e9a6cdf774b0b8cf466162d81675b"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-g0w1"
server_key: "f1789afbc3ac012a16a8c1e58dbdfc1c"
server_key_iv: "546d7dc90b43f360d1f665b7124fd126"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.624904 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 15:06:24.629076 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.630797 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:06:24.630975 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "7b5e9a6cdf774b0b8cf466162d81675b"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-g0w1"
server_key: "f1789afbc3ac012a16a8c1e58dbdfc1c"
server_key_iv: "546d7dc90b43f360d1f665b7124fd126"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.631254 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:24.657344 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:24.661716 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:24.661974 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:24.662918 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:24.664364 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:24.664443 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.664534 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:24.664577 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.698776 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43251
I20260927 15:06:24.698910 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43251 every 8 connection(s)
I20260927 15:06:24.700187 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:24.701176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1159
I20260927 15:06:24.701371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:24.701820 388 external_mini_cluster.cc:1428] Setting key db52b0d1e9862b003c82ebcfa797d636
Sep 27 15:06:24 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:06:24.725342 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:24.703171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38877 (local address 127.0.97.62:37901)
0927 15:06:24.703664 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:24.703671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:24.704943 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 15:06:24.706871 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:24.706879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:24.706977 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:24.707149 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:24.709391 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.710129 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.713068 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.713523 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.713646 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:24.718352 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:24.718358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:24.718361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:24.718398 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:24.721361 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.722319 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.722324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.722326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.722497 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.723058 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.723066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.723096 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.723522 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:24.723689 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:24.724542 (+ 853us) server_negotiation.cc:299] Negotiation successful
0927 15:06:24.724834 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":274,"threads_started":1}
I20260927 15:06:24.726064 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:24.703894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37901 (local address 127.0.97.2:38877)
0927 15:06:24.704753 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:24.704809 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:24.706554 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:24.707424 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:24.707433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:24.708141 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:24.709145 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:24.709176 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.710309 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.710318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:24.712830 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:24.712852 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.713795 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.713805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:24.714184 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:24.715016 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:24.715054 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:24.718041 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:24.721592 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:24.721603 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:24.721629 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:24.722150 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:24.722678 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:24.722682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:24.722685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:24.722904 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:24.723703 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:24.723711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:24.724089 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 15:06:24.724344 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 15:06:24.724797 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":348,"threads_started":1}
I20260927 15:06:24.727900 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37901
I20260927 15:06:24.728250 1287 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:24.729099 1287 heartbeater.cc:511] Master 127.0.97.62:37901 requested a full tablet report, sending...
I20260927 15:06:24.730897 964 ts_manager.cc:194] Registered new tserver with Master: 7b5e9a6cdf774b0b8cf466162d81675b (127.0.97.2:43251)
I20260927 15:06:24.731784 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:24.748018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:25.094230 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:25.094667 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:25.094841 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:25.102391 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:25.102591 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:25.102798 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:25.113171 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:25.114590 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:25.116039 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:25.130059 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:25.130048 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:25.129977 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:25.131034 1293 server_base.cc:1034] running on GCE node
I20260927 15:06:25.131757 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:25.132786 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:25.134018 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790521585133965 us; error 41 us; skew 500 ppm
I20260927 15:06:25.134402 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:25.138099 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:25.139748 1293 webserver.cc:460] Webserver started at http://127.0.97.3:34399/ using document root <none> and password file <none>
I20260927 15:06:25.140475 1293 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:25.140587 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:25.140954 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:25.144066 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "35210c5c96584f76b7d2cf91a6b23976"
format_stamp: "Formatted at 2026-09-27 15:06:25 on dist-test-slave-g0w1"
server_key: "fb6273932e09cbf1eae13e7ccb290ce7"
server_key_iv: "dfcc320b4afc046c297198fb5db07e6f"
server_key_version: "encryptionkey@0"
I20260927 15:06:25.144896 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "35210c5c96584f76b7d2cf91a6b23976"
format_stamp: "Formatted at 2026-09-27 15:06:25 on dist-test-slave-g0w1"
server_key: "fb6273932e09cbf1eae13e7ccb290ce7"
server_key_iv: "dfcc320b4afc046c297198fb5db07e6f"
server_key_version: "encryptionkey@0"
I20260927 15:06:25.150923 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:06:25.155047 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:25.156666 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 15:06:25.156848 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "35210c5c96584f76b7d2cf91a6b23976"
format_stamp: "Formatted at 2026-09-27 15:06:25 on dist-test-slave-g0w1"
server_key: "fb6273932e09cbf1eae13e7ccb290ce7"
server_key_iv: "dfcc320b4afc046c297198fb5db07e6f"
server_key_version: "encryptionkey@0"
I20260927 15:06:25.157061 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:25.179270 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:25.183918 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:25.184234 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:25.185295 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:25.186769 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:25.186847 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:25.186986 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:25.187034 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:25.203539 1153 heartbeater.cc:503] Master 127.0.97.62:37901 was elected leader, sending a full tablet report...
I20260927 15:06:25.222226 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40023
I20260927 15:06:25.222275 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40023 every 8 connection(s)
I20260927 15:06:25.223699 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:25.227219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1293
I20260927 15:06:25.227418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:25.227979 388 external_mini_cluster.cc:1428] Setting key d14859b90423e1dbc0cb1456e10326cd
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:25.246939 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.226662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41761 (local address 127.0.97.62:37901)
0927 15:06:25.227435 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.227440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.228393 (+ 953us) server_negotiation.cc:407] Connection header received
0927 15:06:25.230629 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.230635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.230753 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.230874 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.233126 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.233912 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.235799 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.236228 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.236379 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.240472 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.240477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.240480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.240516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.243721 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.244553 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.244557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.244578 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.244709 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.245207 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.245211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.245213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.245494 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.245640 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.246457 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.246661 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":563,"threads_started":1}
I20260927 15:06:25.248118 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.227379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37901 (local address 127.0.97.3:41761)
0927 15:06:25.228205 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.228250 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:25.230337 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:25.231213 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:25.231224 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:25.231799 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:25.232917 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.232941 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.234104 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.234109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.235605 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.235621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.236498 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.236507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.236784 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.237575 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:25.237601 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:25.240178 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:25.243877 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:25.243894 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:25.243910 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:25.244397 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:25.244849 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:25.244853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:25.244855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:25.245084 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:25.245630 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:25.245638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:25.246028 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 15:06:25.246297 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 15:06:25.246765 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":321,"threads_started":1}
I20260927 15:06:25.249965 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37901
I20260927 15:06:25.250406 1421 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:25.251335 1421 heartbeater.cc:511] Master 127.0.97.62:37901 requested a full tablet report, sending...
I20260927 15:06:25.253154 964 ts_manager.cc:194] Registered new tserver with Master: 35210c5c96584f76b7d2cf91a6b23976 (127.0.97.3:40023)
I20260927 15:06:25.254194 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41761
I20260927 15:06:25.257031 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:25.278179 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.264370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.0.97.62:37901)
0927 15:06:25.264630 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.264635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.265125 (+ 490us) server_negotiation.cc:407] Connection header received
0927 15:06:25.265271 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.265274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.265358 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.265457 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.267140 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.267980 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.269350 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.269835 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.269956 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.272448 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.272453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.272456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.272493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.275333 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.276168 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.276172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.276174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.276320 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.276817 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.276820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.276822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.277177 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.277372 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.277759 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.277938 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:06:25.285511 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:25.293124 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:25.321265 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.306522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40023 (local address 127.0.0.1:33936)
0927 15:06:25.307399 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.307448 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:25.307711 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:25.309264 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:25.309275 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:25.309323 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:25.309900 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.309927 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.313478 (+ 3551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.313489 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.316895 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.316932 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.319777 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.319787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.320117 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.320207 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 15:06:25.320357 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 15:06:25.320655 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":315,"threads_started":1}
I20260927 15:06:25.323375 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.306804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.0.97.3:40023)
0927 15:06:25.308442 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.308451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.308487 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:06:25.308588 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.308596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.308947 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.310194 (+ 1247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:25.310558 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.313233 (+ 2675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.317240 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.319439 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.322077 (+ 2638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.322461 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.322560 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.322827 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":1298,"threads_started":1}
I20260927 15:06:25.329051 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7c4ec090604f59b0351409734976fc (DEFAULT_TABLE table=test-table [id=99ca0e1e6d0a4d6ea0d1baf26c2d3ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:25.330651 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7c4ec090604f59b0351409734976fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:25.332593 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.307595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43251 (local address 127.0.0.1:34288)
0927 15:06:25.321623 (+ 14028us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.321651 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:25.321809 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:25.325505 (+ 3696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:25.325510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:25.325537 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:25.325748 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.325758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.327974 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.327981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.329835 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.329852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.331742 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.331749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.331938 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.331984 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:06:25.332079 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:06:25.332268 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13881,"thread_start_us":17935,"threads_started":1}
I20260927 15:06:25.333518 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.308501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.0.97.2:43251)
0927 15:06:25.324775 (+ 16274us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.324784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.324817 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:06:25.324924 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.324934 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.325245 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.325972 (+ 727us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:25.326230 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.327816 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.330019 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.331550 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.331697 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.331899 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.333008 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.333199 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16123,"thread_start_us":16009,"threads_started":1}
I20260927 15:06:25.334944 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.307774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35631 (local address 127.0.0.1:41626)
0927 15:06:25.325016 (+ 17242us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.325042 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:25.325218 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:25.326603 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:25.326607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:25.326630 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:25.327116 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.327129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.329786 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.329791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.331370 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.331383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.334021 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.334029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.334216 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.334250 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:06:25.334342 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:06:25.334563 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17065,"thread_start_us":16962,"threads_started":1}
I20260927 15:06:25.334959 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.308522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.97.1:35631)
0927 15:06:25.324156 (+ 15634us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.324171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.325769 (+ 1598us) server_negotiation.cc:407] Connection header received
0927 15:06:25.325881 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.325887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.326156 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.326354 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:25.327794 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.329551 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.332015 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.333811 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.333957 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.334176 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.334441 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.334608 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15376,"thread_start_us":15666,"threads_started":1}
I20260927 15:06:25.338007 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7c4ec090604f59b0351409734976fc (DEFAULT_TABLE table=test-table [id=99ca0e1e6d0a4d6ea0d1baf26c2d3ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:25.338536 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7c4ec090604f59b0351409734976fc (DEFAULT_TABLE table=test-table [id=99ca0e1e6d0a4d6ea0d1baf26c2d3ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:25.339589 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7c4ec090604f59b0351409734976fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:25.339972 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7c4ec090604f59b0351409734976fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:25.347400 1440 tablet_bootstrap.cc:492] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: Bootstrap starting.
I20260927 15:06:25.351997 1440 tablet_bootstrap.cc:654] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:25.353437 1441 tablet_bootstrap.cc:492] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: Bootstrap starting.
I20260927 15:06:25.353433 1440 log.cc:824] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:25.354553 1442 tablet_bootstrap.cc:492] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: Bootstrap starting.
I20260927 15:06:25.357626 1441 tablet_bootstrap.cc:654] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:25.358667 1442 tablet_bootstrap.cc:654] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:25.358889 1440 tablet_bootstrap.cc:492] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: No bootstrap required, opened a new log
I20260927 15:06:25.358872 1441 log.cc:824] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:25.359241 1440 ts_tablet_manager.cc:1397] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260927 15:06:25.359978 1442 log.cc:824] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:25.362509 1441 tablet_bootstrap.cc:492] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: No bootstrap required, opened a new log
I20260927 15:06:25.362831 1441 ts_tablet_manager.cc:1397] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 15:06:25.363216 1442 tablet_bootstrap.cc:492] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: No bootstrap required, opened a new log
I20260927 15:06:25.363505 1442 ts_tablet_manager.cc:1397] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260927 15:06:25.369114 1440 raft_consensus.cc:348] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.369570 1440 raft_consensus.cc:374] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:25.369673 1440 raft_consensus.cc:729] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35210c5c96584f76b7d2cf91a6b23976, State: Initialized, Role: FOLLOWER
I20260927 15:06:25.370374 1440 consensus_queue.cc:260] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.371604 1421 heartbeater.cc:503] Master 127.0.97.62:37901 was elected leader, sending a full tablet report...
I20260927 15:06:25.372413 1440 ts_tablet_manager.cc:1428] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260927 15:06:25.374742 1442 raft_consensus.cc:348] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.374742 1441 raft_consensus.cc:348] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.375275 1441 raft_consensus.cc:374] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:25.375275 1442 raft_consensus.cc:374] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:25.375383 1442 raft_consensus.cc:729] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5e9a6cdf774b0b8cf466162d81675b, State: Initialized, Role: FOLLOWER
I20260927 15:06:25.375387 1441 raft_consensus.cc:729] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbc76da85374b25b09c0c3907ce9893, State: Initialized, Role: FOLLOWER
I20260927 15:06:25.376073 1442 consensus_queue.cc:260] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.376078 1441 consensus_queue.cc:260] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.378202 1441 ts_tablet_manager.cc:1428] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260927 15:06:25.378655 1287 heartbeater.cc:503] Master 127.0.97.62:37901 was elected leader, sending a full tablet report...
I20260927 15:06:25.379160 1442 ts_tablet_manager.cc:1428] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260927 15:06:25.419574 1154 tablet.cc:2369] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:06:25.452302 1288 tablet.cc:2369] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:06:25.475505 1422 tablet.cc:2369] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:25.527110 1446 raft_consensus.cc:482] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:25.527508 1446 raft_consensus.cc:504] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.529458 1446 leader_election.cc:288] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cbc76da85374b25b09c0c3907ce9893 (127.0.97.1:35631), 7b5e9a6cdf774b0b8cf466162d81675b (127.0.97.2:43251)
I20260927 15:06:25.537374 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.530059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58725 (local address 127.0.97.2:43251)
0927 15:06:25.530349 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.530358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.530888 (+ 530us) server_negotiation.cc:407] Connection header received
0927 15:06:25.531051 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.531056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.531169 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.531269 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:25.531777 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.532597 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.534979 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.536267 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.536392 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.536478 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.536981 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.537115 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 15:06:25.537457 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.529930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38835 (local address 127.0.97.1:35631)
0927 15:06:25.530125 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.530131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.530151 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:06:25.530375 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.530385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.530478 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.530593 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:25.531041 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.532069 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.534660 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.536154 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.536340 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.536463 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.536964 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.537115 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:06:25.537516 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.529976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43251 (local address 127.0.97.3:58725)
0927 15:06:25.530771 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.530788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:25.530910 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:25.531289 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:25.531292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:25.531311 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:25.531618 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.531627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.532765 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.532771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.534698 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.534708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.536499 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.536505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.536767 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.536792 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:06:25.536884 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:06:25.537104 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":59,"thread_start_us":441,"threads_started":1}
I20260927 15:06:25.537688 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.529657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35631 (local address 127.0.97.3:38835)
0927 15:06:25.530005 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.530028 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:25.530168 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:25.530641 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:25.530646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:25.530675 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:25.530885 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.530897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.532245 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.532249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.534268 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:25.534286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.536364 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.536370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:25.536699 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.536780 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 15:06:25.536884 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:06:25.537096 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":73}
I20260927 15:06:25.538630 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7c4ec090604f59b0351409734976fc" candidate_uuid: "35210c5c96584f76b7d2cf91a6b23976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" is_pre_election: true
I20260927 15:06:25.538667 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7c4ec090604f59b0351409734976fc" candidate_uuid: "35210c5c96584f76b7d2cf91a6b23976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbc76da85374b25b09c0c3907ce9893" is_pre_election: true
I20260927 15:06:25.539211 1108 raft_consensus.cc:2454] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35210c5c96584f76b7d2cf91a6b23976 in term 0.
I20260927 15:06:25.539194 1242 raft_consensus.cc:2454] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35210c5c96584f76b7d2cf91a6b23976 in term 0.
I20260927 15:06:25.539988 1311 leader_election.cc:302] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35210c5c96584f76b7d2cf91a6b23976, 3cbc76da85374b25b09c0c3907ce9893; no voters:
I20260927 15:06:25.540450 1446 raft_consensus.cc:2781] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:25.540642 1446 raft_consensus.cc:482] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:25.540753 1446 raft_consensus.cc:3037] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:25.543488 1446 raft_consensus.cc:504] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.544163 1446 leader_election.cc:288] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [CANDIDATE]: Term 1 election: Requested vote from peers 3cbc76da85374b25b09c0c3907ce9893 (127.0.97.1:35631), 7b5e9a6cdf774b0b8cf466162d81675b (127.0.97.2:43251)
I20260927 15:06:25.544765 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7c4ec090604f59b0351409734976fc" candidate_uuid: "35210c5c96584f76b7d2cf91a6b23976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbc76da85374b25b09c0c3907ce9893"
I20260927 15:06:25.544766 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7c4ec090604f59b0351409734976fc" candidate_uuid: "35210c5c96584f76b7d2cf91a6b23976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5e9a6cdf774b0b8cf466162d81675b"
I20260927 15:06:25.545007 1108 raft_consensus.cc:3037] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:25.545002 1242 raft_consensus.cc:3037] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:25.548290 1108 raft_consensus.cc:2454] T 5a7c4ec090604f59b0351409734976fc P 3cbc76da85374b25b09c0c3907ce9893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35210c5c96584f76b7d2cf91a6b23976 in term 1.
I20260927 15:06:25.548290 1242 raft_consensus.cc:2454] T 5a7c4ec090604f59b0351409734976fc P 7b5e9a6cdf774b0b8cf466162d81675b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35210c5c96584f76b7d2cf91a6b23976 in term 1.
I20260927 15:06:25.548795 1311 leader_election.cc:302] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35210c5c96584f76b7d2cf91a6b23976, 3cbc76da85374b25b09c0c3907ce9893, 7b5e9a6cdf774b0b8cf466162d81675b; no voters:
I20260927 15:06:25.549162 1446 raft_consensus.cc:2781] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:25.549667 1446 raft_consensus.cc:686] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [term 1 LEADER]: Becoming Leader. State: Replica: 35210c5c96584f76b7d2cf91a6b23976, State: Running, Role: LEADER
I20260927 15:06:25.550237 1446 consensus_queue.cc:237] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } }
I20260927 15:06:25.555229 964 catalog_manager.cc:5495] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35210c5c96584f76b7d2cf91a6b23976 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "35210c5c96584f76b7d2cf91a6b23976" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35210c5c96584f76b7d2cf91a6b23976" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cbc76da85374b25b09c0c3907ce9893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:25.630357 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.614819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.0.97.1:35631)
0927 15:06:25.615026 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.615034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.615735 (+ 701us) server_negotiation.cc:407] Connection header received
0927 15:06:25.615848 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.615853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.615947 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.616041 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.617595 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.618358 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.619753 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.620202 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.620311 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.624522 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.624538 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.624556 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.624606 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.627575 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.628392 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.628400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.628403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.628542 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.628946 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.628949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.628951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.629315 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.629536 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.629819 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.629985 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:25.678277 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.664199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.97.1:35631)
0927 15:06:25.664423 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.664432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.664792 (+ 360us) server_negotiation.cc:407] Connection header received
0927 15:06:25.665008 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.665012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.665077 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.665186 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.666538 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.667379 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.668660 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.669091 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.669231 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.673237 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.673242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.673245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.673284 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.675920 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.676593 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.676596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.676599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.676730 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.677119 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.677121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.677124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.677378 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.677600 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.677846 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.678021 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:25.725935 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.711195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.97.1:35631)
0927 15:06:25.711432 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.711439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.711740 (+ 301us) server_negotiation.cc:407] Connection header received
0927 15:06:25.711957 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.711967 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.712032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.712137 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.713647 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.714369 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.715748 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.716209 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.716317 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.720351 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.720357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.720360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.720396 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.722994 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.723914 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.723919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.723922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.724059 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.724514 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.724518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.724520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.724867 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.725143 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.725368 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.725563 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 15:06:25.727700 1068 tablet_service.cc:2908] Rejecting scan request for tablet 5a7c4ec090604f59b0351409734976fc: Uninitialized: clean time has not yet been initialized
I20260927 15:06:25.731062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1025
I20260927 15:06:25.745478 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.731944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.97.1:35631)
0927 15:06:25.732107 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.732112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.732470 (+ 358us) server_negotiation.cc:407] Connection header received
0927 15:06:25.732634 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.732638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.732709 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.732792 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.733961 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.734638 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.736019 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.736434 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.736589 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.739521 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.739528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.739532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.739581 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.742638 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.743465 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.743470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.743474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.743609 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.743990 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.743995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.744008 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.744292 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.744597 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.744903 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.745144 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:06:25.746157 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:25.837215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1025
I20260927 15:06:25.854550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1159
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:25.871559 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.855535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.0.97.2:43251)
0927 15:06:25.855707 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.855711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.855729 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:06:25.855827 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.855831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.855924 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.856010 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.857525 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.858449 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.859995 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.860536 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.860646 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.864897 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.864916 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.864932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.864973 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.868160 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.869082 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.869091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.869094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.869244 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.869715 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.869721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.869724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.870168 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:25.870489 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:25.870852 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 15:06:25.871195 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:06:25.872452 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:25.967666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1159
I20260927 15:06:25.967625 1446 consensus_queue.cc:1035] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5e9a6cdf774b0b8cf466162d81675b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:06:25.984647 1312 connection.cc:535] client connection to 127.0.97.2:43251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 15:06:25.984954 1312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:06:25.985056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1293
I20260927 15:06:25.986724 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.986203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43251 (local address 127.0.97.3:56787)
0927 15:06:25.986389 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:25.986511 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 15:06:25.986966 1312 consensus_peers.cc:487] T 5a7c4ec090604f59b0351409734976fc P 35210c5c96584f76b7d2cf91a6b23976 -> Peer 7b5e9a6cdf774b0b8cf466162d81675b (127.0.97.2:43251): Couldn't send request to peer 7b5e9a6cdf774b0b8cf466162d81675b. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.2:43251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 27 15:06:25 dist-test-slave-g0w1 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:26.001201 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:25.985845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.97.3:40023)
0927 15:06:25.986336 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:25.986342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:25.986364 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:06:25.986423 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:25.986428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:25.986508 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:25.986617 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:25.988094 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.988822 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.990082 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:25.990598 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:25.990727 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:25.995269 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:25.995283 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:25.995300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:25.995340 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:25.998343 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.999165 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.999176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.999178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:25.999330 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:25.999738 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:25.999741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:25.999744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:26.000143 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:26.000367 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:26.000591 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:06:26.000792 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":463,"threads_started":1}
I20260927 15:06:26.001883 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:26.093403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1293
I20260927 15:06:26.109970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 932
I20260927 15:06:26.111557 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:26.219601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 932
2026-09-27T15:06:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4467 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:26 dist-test-slave-g0w1 krb5kdc[1471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:26 dist-test-slave-g0w1 krb5kdc[1471](info): set up 2 sockets
Sep 27 15:06:26 dist-test-slave-g0w1 krb5kdc[1471](info): commencing operation
krb5kdc: starting...
W20260927 15:06:27.311435 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:27 dist-test-slave-g0w1 krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:27.507705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46729
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:27.848096 1484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:27.848423 1484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:27.848604 1484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:27.856560 1484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:27.856712 1484 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:06:27.856799 1484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:27.856902 1484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:27.856966 1484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:27.857056 1484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:27.868754 1484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46729
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:27.870272 1484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:27.871860 1484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:27.884964 1489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:27.885056 1490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:27.885084 1492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:27.886142 1484 server_base.cc:1034] running on GCE node
I20260927 15:06:27.887310 1484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:27.889725 1484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:27.891037 1484 hybrid_clock.cc:648] HybridClock initialized: now 1790521587890934 us; error 68 us; skew 500 ppm
I20260927 15:06:27.891506 1484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:27 dist-test-slave-g0w1 krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:27.895267 1484 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:27.897059 1484 webserver.cc:460] Webserver started at http://127.0.97.62:39263/ using document root <none> and password file <none>
I20260927 15:06:27.897852 1484 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:27.897964 1484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:27.898355 1484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:27.901526 1484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "8ec75f43020a4be1925796540f74fa9a"
format_stamp: "Formatted at 2026-09-27 15:06:27 on dist-test-slave-g0w1"
server_key: "1e8307f2685ab08b9ecc18976b3280a9"
server_key_iv: "762f7894f970821676169c6060730db1"
server_key_version: "encryptionkey@0"
I20260927 15:06:27.902377 1484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ec75f43020a4be1925796540f74fa9a"
format_stamp: "Formatted at 2026-09-27 15:06:27 on dist-test-slave-g0w1"
server_key: "1e8307f2685ab08b9ecc18976b3280a9"
server_key_iv: "762f7894f970821676169c6060730db1"
server_key_version: "encryptionkey@0"
I20260927 15:06:27.907940 1484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:06:27.911924 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:27.913473 1484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:27.913682 1484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "8ec75f43020a4be1925796540f74fa9a"
format_stamp: "Formatted at 2026-09-27 15:06:27 on dist-test-slave-g0w1"
server_key: "1e8307f2685ab08b9ecc18976b3280a9"
server_key_iv: "762f7894f970821676169c6060730db1"
server_key_version: "encryptionkey@0"
I20260927 15:06:27.913916 1484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:27.930343 1484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:27.935429 1484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:27.935837 1484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:27.958946 1484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46729
I20260927 15:06:27.958899 1560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46729 every 8 connection(s)
I20260927 15:06:27.960435 1484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:27.965955 1561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:27.966516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1484
I20260927 15:06:27.966784 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:27.967549 388 external_mini_cluster.cc:1428] Setting key 34a92dd842709aa1b4e632bd4118aa83
I20260927 15:06:27.979274 1561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: Bootstrap starting.
Sep 27 15:06:27 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:27.982755 1561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:27.984102 1561 log.cc:824] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:27.987275 1561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: No bootstrap required, opened a new log
I20260927 15:06:27.990782 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:27.969758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.97.62:46729)
0927 15:06:27.970603 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:27.970617 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:27.970669 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:06:27.972645 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:27.972675 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:27.973157 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:27.973664 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:27.975143 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:27.976486 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:27.977761 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:27.978491 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:27.978692 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:27.983041 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:27.983050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:27.983124 (+ 74us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:27.983177 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:27.986405 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:27.987224 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:27.987237 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:27.987244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:27.987476 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:27.987878 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:27.987884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:27.987887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:27.988483 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:27.988789 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:27.989279 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 15:06:27.989728 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":388,"threads_started":1}
I20260927 15:06:27.995782 1561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec75f43020a4be1925796540f74fa9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46729 } }
I20260927 15:06:27.996170 1561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:27.996299 1561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec75f43020a4be1925796540f74fa9a, State: Initialized, Role: FOLLOWER
I20260927 15:06:27.996996 1561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec75f43020a4be1925796540f74fa9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46729 } }
I20260927 15:06:27.997326 1561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:27.997470 1561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:27.997646 1561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:27.999969 1561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec75f43020a4be1925796540f74fa9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46729 } }
I20260927 15:06:28.000526 1561 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec75f43020a4be1925796540f74fa9a; no voters:
I20260927 15:06:28.001132 1561 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:28.001276 1566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:28.002933 1566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [term 1 LEADER]: Becoming Leader. State: Replica: 8ec75f43020a4be1925796540f74fa9a, State: Running, Role: LEADER
I20260927 15:06:28.003537 1561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:28.003497 1566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec75f43020a4be1925796540f74fa9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46729 } }
I20260927 15:06:28.006304 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec75f43020a4be1925796540f74fa9a. Latest consensus state: current_term: 1 leader_uuid: "8ec75f43020a4be1925796540f74fa9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec75f43020a4be1925796540f74fa9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46729 } } }
I20260927 15:06:28.006630 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:28.006923 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec75f43020a4be1925796540f74fa9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec75f43020a4be1925796540f74fa9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46729 } } }
I20260927 15:06:28.007323 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:28.008759 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:28.015210 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:28.025231 1575 catalog_manager.cc:1347] Generated new cluster ID: 1f9433af56014d6b99324367ce871b04
I20260927 15:06:28.025372 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:28.039965 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:28.041392 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:28.051277 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: Generated new TSK 0
I20260927 15:06:28.052234 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:28.103359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46729
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:28.449508 1587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:28.449812 1587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:28.449992 1587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:28.457425 1587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:28.457573 1587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:28.457662 1587 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:06:28.457756 1587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:28.467865 1587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:28.469216 1587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:28.470669 1587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:28.484486 1592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:28.484521 1593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:28.484575 1595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:28.485596 1587 server_base.cc:1034] running on GCE node
I20260927 15:06:28.486359 1587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:28.487457 1587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:28.488695 1587 hybrid_clock.cc:648] HybridClock initialized: now 1790521588488626 us; error 43 us; skew 500 ppm
I20260927 15:06:28.489001 1587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:28 dist-test-slave-g0w1 krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:28.498245 1587 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:28.500078 1587 webserver.cc:460] Webserver started at http://127.0.97.1:42185/ using document root <none> and password file <none>
I20260927 15:06:28.500893 1587 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:28.500994 1587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:28.501344 1587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:28.504652 1587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "889b98be65044941b1904d940eb4956c"
format_stamp: "Formatted at 2026-09-27 15:06:28 on dist-test-slave-g0w1"
server_key: "6f11eb781bbf216f394ac76bb66a6be8"
server_key_iv: "dfd71130d64217dafc0fa7edfd2aefc6"
server_key_version: "encryptionkey@0"
I20260927 15:06:28.505584 1587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "889b98be65044941b1904d940eb4956c"
format_stamp: "Formatted at 2026-09-27 15:06:28 on dist-test-slave-g0w1"
server_key: "6f11eb781bbf216f394ac76bb66a6be8"
server_key_iv: "dfd71130d64217dafc0fa7edfd2aefc6"
server_key_version: "encryptionkey@0"
I20260927 15:06:28.511323 1587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:06:28.515244 1602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:28.516825 1587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:28.517046 1587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "889b98be65044941b1904d940eb4956c"
format_stamp: "Formatted at 2026-09-27 15:06:28 on dist-test-slave-g0w1"
server_key: "6f11eb781bbf216f394ac76bb66a6be8"
server_key_iv: "dfd71130d64217dafc0fa7edfd2aefc6"
server_key_version: "encryptionkey@0"
I20260927 15:06:28.517304 1587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:28.532181 1587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:28.536792 1587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:28.537098 1587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:28.540110 1587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:28.540221 1587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:28.540349 1587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:28.540429 1587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:06:28 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:28.567634 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:28.540734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40003 (local address 127.0.97.62:46729)
0927 15:06:28.541344 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:28.541349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:28.542564 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 15:06:28.544714 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:28.544720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:28.544819 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:28.544949 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:28.547975 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.549020 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.552487 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.553062 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.553234 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:28.559007 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:28.559019 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:28.559027 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:28.559107 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:28.563441 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.564875 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.564884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.564891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.565089 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.565685 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.565690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.565694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.566002 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:28.566182 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:28.566948 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 15:06:28.567243 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":401,"threads_started":1}
I20260927 15:06:28.569618 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:28.541446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.97.1:40003)
0927 15:06:28.542366 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:28.542410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:28.544389 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:28.545314 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:28.545332 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:28.546452 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:28.547625 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:28.547662 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.549230 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.549239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:28.552229 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:28.552281 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.553509 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.553515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:28.553717 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:28.554662 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:28.554687 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:28.558580 (+ 3893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:28.563695 (+ 5115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:28.563709 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:28.563740 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:28.564648 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:28.565234 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:28.565245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:28.565249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:28.565532 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:28.566160 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:28.566169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:28.566557 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 15:06:28.566848 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 15:06:28.567493 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":393,"threads_started":1}
I20260927 15:06:28.587314 1587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40201
I20260927 15:06:28.587332 1722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40201 every 8 connection(s)
I20260927 15:06:28.588737 1587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:28.591359 1723 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46729
I20260927 15:06:28.591420 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:28.576756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33771 (local address 127.0.97.62:46729)
0927 15:06:28.576952 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:28.576957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:28.577259 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:06:28.577507 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:28.577514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:28.577606 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:28.577723 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:28.578985 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.580038 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.582291 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.582720 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.582865 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:28.585347 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:28.585353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:28.585356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:28.585392 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:28.588243 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.589055 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.589061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.589067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.589207 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.589672 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.589679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.589682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.590062 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:28.590205 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:28.590781 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 15:06:28.591019 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:06:28.591605 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:28.576361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.97.1:33771)
0927 15:06:28.577122 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:28.577150 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:28.577328 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:28.577898 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:28.577902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:28.578419 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:28.578811 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:28.578824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.580182 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.580188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:28.582093 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:28.582112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.582932 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.582941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:28.583173 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:28.583932 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:28.583964 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:28.585109 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:28.588408 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:28.588416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:28.588422 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:28.588922 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:28.589330 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:28.589335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:28.589339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:28.589506 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:28.590309 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:28.590312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:28.590533 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 15:06:28.590687 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 15:06:28.591097 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":418,"threads_started":1}
I20260927 15:06:28.591666 1723 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:28.592392 1723 heartbeater.cc:511] Master 127.0.97.62:46729 requested a full tablet report, sending...
I20260927 15:06:28.593037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1587
I20260927 15:06:28.593267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:28.593986 388 external_mini_cluster.cc:1428] Setting key 453bc15231950b451360ed419c4041c2
I20260927 15:06:28.595144 1516 ts_manager.cc:194] Registered new tserver with Master: 889b98be65044941b1904d940eb4956c (127.0.97.1:40201)
I20260927 15:06:28.596781 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40003
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:28.636554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46729
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:28.989669 1727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:28.990020 1727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:28.990191 1727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:28.997833 1727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:28.997999 1727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:28.998159 1727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:06:28.998297 1727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:29.008952 1727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:29.010342 1727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:29.011816 1727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:29.025908 1735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:29.025871 1732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:29.025947 1733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:29.026981 1727 server_base.cc:1034] running on GCE node
I20260927 15:06:29.027689 1727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:29.028714 1727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:29.030001 1727 hybrid_clock.cc:648] HybridClock initialized: now 1790521589029927 us; error 46 us; skew 500 ppm
I20260927 15:06:29.030373 1727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:29 dist-test-slave-g0w1 krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:29.033802 1727 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:29.035561 1727 webserver.cc:460] Webserver started at http://127.0.97.2:45035/ using document root <none> and password file <none>
I20260927 15:06:29.036343 1727 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:29.036489 1727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:29.036835 1727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:29.040066 1727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "d03cb17ff3c24fd09dac021fcf624aad"
format_stamp: "Formatted at 2026-09-27 15:06:29 on dist-test-slave-g0w1"
server_key: "7eff201a1fb891f5193eeb3347166db9"
server_key_iv: "6c2a4d374b0160f96fd6e054f82f88eb"
server_key_version: "encryptionkey@0"
I20260927 15:06:29.040967 1727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d03cb17ff3c24fd09dac021fcf624aad"
format_stamp: "Formatted at 2026-09-27 15:06:29 on dist-test-slave-g0w1"
server_key: "7eff201a1fb891f5193eeb3347166db9"
server_key_iv: "6c2a4d374b0160f96fd6e054f82f88eb"
server_key_version: "encryptionkey@0"
I20260927 15:06:29.046788 1727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:29.050964 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:29.052644 1727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:29.052847 1727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "d03cb17ff3c24fd09dac021fcf624aad"
format_stamp: "Formatted at 2026-09-27 15:06:29 on dist-test-slave-g0w1"
server_key: "7eff201a1fb891f5193eeb3347166db9"
server_key_iv: "6c2a4d374b0160f96fd6e054f82f88eb"
server_key_version: "encryptionkey@0"
I20260927 15:06:29.053112 1727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:29.087378 1727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:29.092617 1727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:29.092934 1727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:29.096658 1727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:29.096783 1727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:29.096904 1727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:29.096972 1727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:06:29 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:29.122023 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.096994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43691 (local address 127.0.97.62:46729)
0927 15:06:29.097484 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.097489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.098945 (+ 1456us) server_negotiation.cc:407] Connection header received
0927 15:06:29.101143 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.101150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.101247 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.101383 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:29.103986 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.104903 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.107897 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.108670 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.108845 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.114697 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:29.114709 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:29.114715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:29.114778 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:29.118503 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.119485 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.119491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.119494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.119638 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.120112 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.120117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.120121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.120481 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:29.120637 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.121399 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.121632 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":284,"threads_started":1}
I20260927 15:06:29.123239 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.097865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.97.2:43691)
0927 15:06:29.098757 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.098809 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.100815 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.101653 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.101665 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.102499 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:29.103761 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.103792 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.105078 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.105087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.107602 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.107618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.108938 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.108951 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.109339 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.110293 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:29.110332 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:29.114272 (+ 3940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:29.118690 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.118703 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.118725 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.119323 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.119757 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.119762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.119766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.119984 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.120610 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:29.120619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:29.121014 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.121294 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.121908 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":385,"threads_started":1}
I20260927 15:06:29.148931 1727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42903
I20260927 15:06:29.149022 1862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42903 every 8 connection(s)
I20260927 15:06:29.150354 1727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:29.152769 1863 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46729
I20260927 15:06:29.153054 1863 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:29.153874 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.131650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.97.2:43173)
0927 15:06:29.135874 (+ 4224us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.135908 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.136131 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.136741 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.136748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.137703 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:29.138325 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.138345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.139764 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.139770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.142117 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.142157 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.143244 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.143255 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.143497 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.144426 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:29.144460 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:29.146106 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:29.150180 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.150186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.150191 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.150757 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.151359 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.151367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.151373 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.151591 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.152795 (+ 1204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:29.152798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:29.153030 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.153244 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.153532 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3977,"thread_start_us":3772,"threads_started":1}
I20260927 15:06:29.153949 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.131643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43173 (local address 127.0.97.62:46729)
0927 15:06:29.131878 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.131887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.136060 (+ 4173us) server_negotiation.cc:407] Connection header received
0927 15:06:29.136321 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.136330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.136445 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.136596 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:29.138564 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.139565 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.142439 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.142949 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.143145 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.146451 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:29.146461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:29.146469 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:29.146523 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:29.150011 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.150968 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.150978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.150983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.151209 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.151743 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.151747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.151750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.152286 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:29.152718 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.153383 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.153632 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:06:29.154001 1863 heartbeater.cc:511] Master 127.0.97.62:46729 requested a full tablet report, sending...
I20260927 15:06:29.155530 1516 ts_manager.cc:194] Registered new tserver with Master: d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903)
I20260927 15:06:29.156394 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43691
I20260927 15:06:29.157294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1727
I20260927 15:06:29.157493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:29.158025 388 external_mini_cluster.cc:1428] Setting key 54d50a303592bbdf3314c1196d3c4793
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:29.199747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46729
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:29.545281 1867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:29.545593 1867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:29.545696 1867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:29.553261 1867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:29.553392 1867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:29.553478 1867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:06:29.553550 1867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:29.564947 1867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:29.566258 1867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:29.568078 1867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:29.583287 1872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:29.583320 1873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:29.583405 1875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:29.584430 1867 server_base.cc:1034] running on GCE node
I20260927 15:06:29.585332 1867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:29.586686 1867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:29.588008 1867 hybrid_clock.cc:648] HybridClock initialized: now 1790521589587888 us; error 94 us; skew 500 ppm
I20260927 15:06:29.588438 1867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:29 dist-test-slave-g0w1 krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:29.592934 1867 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:29.594805 1867 webserver.cc:460] Webserver started at http://127.0.97.3:42271/ using document root <none> and password file <none>
I20260927 15:06:29.595661 1867 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:29.595791 1867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:29.596144 1867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:29.599532 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab"
format_stamp: "Formatted at 2026-09-27 15:06:29 on dist-test-slave-g0w1"
server_key: "9974e8b2a1ed819a6f642cbf66d70104"
server_key_iv: "3de0e988b97f92bc355140f07d4a2056"
server_key_version: "encryptionkey@0"
I20260927 15:06:29.600415 1723 heartbeater.cc:503] Master 127.0.97.62:46729 was elected leader, sending a full tablet report...
I20260927 15:06:29.600436 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab"
format_stamp: "Formatted at 2026-09-27 15:06:29 on dist-test-slave-g0w1"
server_key: "9974e8b2a1ed819a6f642cbf66d70104"
server_key_iv: "3de0e988b97f92bc355140f07d4a2056"
server_key_version: "encryptionkey@0"
I20260927 15:06:29.606719 1867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:06:29.610944 1882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:29.612776 1867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:29.613005 1867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab"
format_stamp: "Formatted at 2026-09-27 15:06:29 on dist-test-slave-g0w1"
server_key: "9974e8b2a1ed819a6f642cbf66d70104"
server_key_iv: "3de0e988b97f92bc355140f07d4a2056"
server_key_version: "encryptionkey@0"
I20260927 15:06:29.613260 1867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:29.640144 1867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:29.645174 1867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:29.645493 1867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:29.648543 1867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:29.648671 1867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:29.648816 1867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:29.648916 1867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:06:29 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:29.676640 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.649296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59649 (local address 127.0.97.62:46729)
0927 15:06:29.649502 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.649508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.651153 (+ 1645us) server_negotiation.cc:407] Connection header received
0927 15:06:29.653569 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.653579 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.653705 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.653840 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:29.656481 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.657677 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.661360 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.662020 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.662183 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.668519 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:29.668525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:29.668528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:29.668566 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:29.672110 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.673297 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.673303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.673308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.673467 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.674113 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.674122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.674124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.674402 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:29.674584 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.675755 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.676107 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:06:29.677296 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.649973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.97.3:59649)
0927 15:06:29.650925 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.650993 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.653114 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.654144 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.654159 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.655159 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:29.656215 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.656266 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.657894 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.657907 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.661132 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.661159 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.662332 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.662343 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.662760 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.663752 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:29.663792 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:29.668148 (+ 4356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:29.672276 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.672286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.672309 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.673141 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.673594 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.673599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.673603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.673970 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.674552 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:29.674561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:29.675200 (+ 639us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.675573 (+ 373us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.676068 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":412,"threads_started":1}
I20260927 15:06:29.699584 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.683941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40065 (local address 127.0.97.62:46729)
0927 15:06:29.684184 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.684193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.684815 (+ 622us) server_negotiation.cc:407] Connection header received
0927 15:06:29.684960 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.684968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.685081 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.685216 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:29.686179 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.687602 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.689823 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.690235 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.690383 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.693162 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:29.693167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:29.693170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:29.693203 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:29.696910 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.697608 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.697614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.697618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.697763 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.698131 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.698137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.698141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.698414 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:29.698571 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.698963 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.699205 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:06:29.699795 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.684043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.97.3:40065)
0927 15:06:29.684698 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.684723 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.684848 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.685253 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.685256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.685733 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:29.686026 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.686037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.687807 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.687815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.689633 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.689649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.690433 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.690443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.690630 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.691468 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:29.691497 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:29.692918 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:29.697079 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.697084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.697086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.697474 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.697884 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:29.697889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:29.697894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:29.698022 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:29.698532 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:29.698537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:29.698721 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.698867 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.699252 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":356,"threads_started":1}
I20260927 15:06:29.703949 1867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40383
I20260927 15:06:29.703980 2001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40383 every 8 connection(s)
I20260927 15:06:29.705374 1867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:29.707767 2002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46729
I20260927 15:06:29.708012 2002 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:29.708712 2002 heartbeater.cc:511] Master 127.0.97.62:46729 requested a full tablet report, sending...
I20260927 15:06:29.710114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 1867
I20260927 15:06:29.710316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:29.710283 1516 ts_manager.cc:194] Registered new tserver with Master: 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3:40383)
I20260927 15:06:29.710747 388 external_mini_cluster.cc:1428] Setting key b35ec2988bc7abb0454e06954cfd2b2e
I20260927 15:06:29.711238 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59649
I20260927 15:06:29.714107 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:06:29 dist-test-slave-g0w1 krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:06:29 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:29.763829 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.747935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.0.97.62:46729)
0927 15:06:29.748162 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.748169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.748369 (+ 200us) server_negotiation.cc:407] Connection header received
0927 15:06:29.748628 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.748632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.748702 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.748818 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:29.750286 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.751040 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.752394 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.752818 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.752971 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.758298 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:29.758304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:29.758308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:29.758350 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:29.761215 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.761941 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.761945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.761948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.762099 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:29.762478 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:29.762483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:29.762486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:29.762804 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:29.763032 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.763370 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.763586 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:06:29.768692 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:29.775969 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:29.825898 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.789882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.0.97.1:40201)
0927 15:06:29.815967 (+ 26085us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.815977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.817404 (+ 1427us) server_negotiation.cc:407] Connection header received
0927 15:06:29.817692 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.817703 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.817978 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.818266 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:29.818964 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.820296 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.823651 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.824637 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.824798 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.825029 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.825376 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.825585 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25944,"thread_start_us":26097,"threads_started":1}
I20260927 15:06:29.826292 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.789918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.0.97.2:42903)
0927 15:06:29.816849 (+ 26931us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.816858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.816882 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:06:29.816956 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.816962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.817230 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.817446 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:29.818419 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.820025 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.823404 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.824839 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.824969 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.825166 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.825728 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.825892 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26763,"thread_start_us":26627,"threads_started":1}
I20260927 15:06:29.826804 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.790109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.97.3:40383)
0927 15:06:29.815753 (+ 25644us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:29.815765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:29.816813 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 15:06:29.816928 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:29.816936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:29.817231 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:29.817484 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:29.818694 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.820984 (+ 2290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.823320 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.824839 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.824969 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.825194 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:29.826308 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0927 15:06:29.826475 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25431,"thread_start_us":25675,"threads_started":1}
I20260927 15:06:29.827184 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.789752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40383 (local address 127.0.0.1:34702)
0927 15:06:29.816476 (+ 26724us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.816501 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.816676 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.817795 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.817801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.817833 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:29.818367 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.818378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.821152 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.821157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.823139 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.823157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.825243 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.825252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.825434 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.825467 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.825570 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.825776 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26559,"mutex_wait_us":29,"thread_start_us":26573,"threads_started":1}
I20260927 15:06:29.827143 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.789451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42903 (local address 127.0.0.1:59642)
0927 15:06:29.814257 (+ 24806us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.814336 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.814595 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.817537 (+ 2942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.817546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.817661 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:29.818173 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.818190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.820180 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.820187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.823254 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.823269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.825370 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.825375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.825538 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.825574 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.825672 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.825812 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24411,"mutex_wait_us":112,"thread_start_us":28292,"threads_started":1}
I20260927 15:06:29.827718 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:29.789469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40201 (local address 127.0.0.1:34338)
0927 15:06:29.817262 (+ 27793us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:29.817279 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:29.817438 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:29.818349 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:29.818355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:29.818378 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:29.818804 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.818816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.820422 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.820428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.823474 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:29.823483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:29.824845 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:29.824851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:29.825043 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:29.825105 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 15:06:29.825252 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:06:29.826265 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27699,"mutex_wait_us":642,"thread_start_us":24786,"threads_started":1}
I20260927 15:06:29.830945 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 272bdaa01d0a445eb6a8205705c92214 (DEFAULT_TABLE table=test-table [id=78d2cb4f27a44ee390cc1d3a2182ca0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:29.831162 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 272bdaa01d0a445eb6a8205705c92214 (DEFAULT_TABLE table=test-table [id=78d2cb4f27a44ee390cc1d3a2182ca0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:29.831278 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 272bdaa01d0a445eb6a8205705c92214 (DEFAULT_TABLE table=test-table [id=78d2cb4f27a44ee390cc1d3a2182ca0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:29.832368 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272bdaa01d0a445eb6a8205705c92214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:29.832517 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272bdaa01d0a445eb6a8205705c92214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:29.832541 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272bdaa01d0a445eb6a8205705c92214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:29.844501 2021 tablet_bootstrap.cc:492] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Bootstrap starting.
I20260927 15:06:29.848448 2021 tablet_bootstrap.cc:654] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:29.848512 2022 tablet_bootstrap.cc:492] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Bootstrap starting.
I20260927 15:06:29.849995 2021 log.cc:824] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:29.852638 2023 tablet_bootstrap.cc:492] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Bootstrap starting.
I20260927 15:06:29.853170 2022 tablet_bootstrap.cc:654] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:29.854408 2021 tablet_bootstrap.cc:492] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: No bootstrap required, opened a new log
I20260927 15:06:29.854552 2022 log.cc:824] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:29.854743 2021 ts_tablet_manager.cc:1397] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260927 15:06:29.856535 2023 tablet_bootstrap.cc:654] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:29.857940 2023 log.cc:824] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:29.858037 2022 tablet_bootstrap.cc:492] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: No bootstrap required, opened a new log
I20260927 15:06:29.858325 2022 ts_tablet_manager.cc:1397] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 15:06:29.864364 2021 raft_consensus.cc:348] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:29.864804 2021 raft_consensus.cc:374] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:29.864897 2021 raft_consensus.cc:729] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf130632afc44e7a5a8fe6b0ff5e6ab, State: Initialized, Role: FOLLOWER
I20260927 15:06:29.865550 2021 consensus_queue.cc:260] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:29.866454 2023 tablet_bootstrap.cc:492] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: No bootstrap required, opened a new log
I20260927 15:06:29.866786 2023 ts_tablet_manager.cc:1397] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 15:06:29.866791 2002 heartbeater.cc:503] Master 127.0.97.62:46729 was elected leader, sending a full tablet report...
I20260927 15:06:29.867560 2021 ts_tablet_manager.cc:1428] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Time spent starting tablet: real 0.013s user 0.001s sys 0.014s
I20260927 15:06:29.870977 2022 raft_consensus.cc:348] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:29.871543 2022 raft_consensus.cc:374] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:29.871651 2022 raft_consensus.cc:729] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889b98be65044941b1904d940eb4956c, State: Initialized, Role: FOLLOWER
I20260927 15:06:29.872337 2022 consensus_queue.cc:260] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:29.874285 2022 ts_tablet_manager.cc:1428] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260927 15:06:29.877172 2023 raft_consensus.cc:348] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:29.877702 2023 raft_consensus.cc:374] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:29.877810 2023 raft_consensus.cc:729] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03cb17ff3c24fd09dac021fcf624aad, State: Initialized, Role: FOLLOWER
I20260927 15:06:29.878407 2023 consensus_queue.cc:260] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:29.879580 1863 heartbeater.cc:503] Master 127.0.97.62:46729 was elected leader, sending a full tablet report...
I20260927 15:06:29.880172 2023 ts_tablet_manager.cc:1428] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Time spent starting tablet: real 0.013s user 0.005s sys 0.010s
W20260927 15:06:29.903334 1864 tablet.cc:2369] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:06:29.957355 2003 tablet.cc:2369] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:06:30.091661 1724 tablet.cc:2369] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:30.123046 2029 raft_consensus.cc:482] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:30.123502 2029 raft_consensus.cc:504] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.125247 2029 leader_election.cc:288] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889b98be65044941b1904d940eb4956c (127.0.97.1:40201), 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3:40383)
I20260927 15:06:30.132850 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.125825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37379 (local address 127.0.97.3:40383)
0927 15:06:30.126021 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.126030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.126876 (+ 846us) server_negotiation.cc:407] Connection header received
0927 15:06:30.127069 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.127074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.127223 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.127330 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:30.127949 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.128708 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.130948 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.131789 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.131896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.132043 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.132466 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.132592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:30.132851 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.125608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35993 (local address 127.0.97.1:40201)
0927 15:06:30.125828 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.125835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.126692 (+ 857us) server_negotiation.cc:407] Connection header received
0927 15:06:30.126872 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.126878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.126970 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.127111 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:30.127785 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.128565 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.130918 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.131815 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.131984 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.132089 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.132439 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.132570 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:06:30.133038 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.125646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40201 (local address 127.0.97.2:35993)
0927 15:06:30.126567 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.126584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.126727 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.127129 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.127133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.127156 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:30.127614 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.127628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.128802 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.128809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.130742 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.130757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.132076 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.132082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.132256 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.132294 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.132371 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.132547 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":7,"thread_start_us":519,"threads_started":1}
I20260927 15:06:30.133090 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.125802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40383 (local address 127.0.97.2:37379)
0927 15:06:30.126742 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.126760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.126917 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.127353 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.127356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.127375 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:30.127764 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.127777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.128872 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.128876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.130758 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.130771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.132038 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.132045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.132268 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.132304 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.132423 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.132558 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":43,"thread_start_us":548,"threads_started":1}
I20260927 15:06:30.134142 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272bdaa01d0a445eb6a8205705c92214" candidate_uuid: "d03cb17ff3c24fd09dac021fcf624aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889b98be65044941b1904d940eb4956c" is_pre_election: true
I20260927 15:06:30.134375 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272bdaa01d0a445eb6a8205705c92214" candidate_uuid: "d03cb17ff3c24fd09dac021fcf624aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" is_pre_election: true
I20260927 15:06:30.134790 1673 raft_consensus.cc:2454] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03cb17ff3c24fd09dac021fcf624aad in term 0.
I20260927 15:06:30.134845 1952 raft_consensus.cc:2454] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03cb17ff3c24fd09dac021fcf624aad in term 0.
I20260927 15:06:30.135527 1745 leader_election.cc:302] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 889b98be65044941b1904d940eb4956c, 8bf130632afc44e7a5a8fe6b0ff5e6ab, d03cb17ff3c24fd09dac021fcf624aad; no voters:
I20260927 15:06:30.135987 2029 raft_consensus.cc:2781] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:30.136149 2029 raft_consensus.cc:482] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:30.136258 2029 raft_consensus.cc:3037] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.138538 2029 raft_consensus.cc:504] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.139336 2029 leader_election.cc:288] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [CANDIDATE]: Term 1 election: Requested vote from peers 889b98be65044941b1904d940eb4956c (127.0.97.1:40201), 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3:40383)
I20260927 15:06:30.139889 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272bdaa01d0a445eb6a8205705c92214" candidate_uuid: "d03cb17ff3c24fd09dac021fcf624aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889b98be65044941b1904d940eb4956c"
I20260927 15:06:30.139889 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272bdaa01d0a445eb6a8205705c92214" candidate_uuid: "d03cb17ff3c24fd09dac021fcf624aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab"
I20260927 15:06:30.140121 1952 raft_consensus.cc:3037] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.140128 1673 raft_consensus.cc:3037] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.143877 1673 raft_consensus.cc:2454] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03cb17ff3c24fd09dac021fcf624aad in term 1.
I20260927 15:06:30.143877 1952 raft_consensus.cc:2454] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03cb17ff3c24fd09dac021fcf624aad in term 1.
I20260927 15:06:30.144464 1745 leader_election.cc:302] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889b98be65044941b1904d940eb4956c, d03cb17ff3c24fd09dac021fcf624aad; no voters:
I20260927 15:06:30.144821 2029 raft_consensus.cc:2781] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:30.145341 2029 raft_consensus.cc:686] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 1 LEADER]: Becoming Leader. State: Replica: d03cb17ff3c24fd09dac021fcf624aad, State: Running, Role: LEADER
I20260927 15:06:30.145852 2029 consensus_queue.cc:237] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.151002 1516 catalog_manager.cc:5495] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad reported cstate change: term changed from 0 to 1, leader changed from <none> to d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d03cb17ff3c24fd09dac021fcf624aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } health_report { overall_health: UNKNOWN } } }
I20260927 15:06:30.183604 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.177265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.97.2:42903)
0927 15:06:30.177473 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.177477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.177518 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:06:30.177914 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.177918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.177992 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.178142 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:30.178510 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.179304 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.180793 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.181613 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.181756 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.182295 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:30.182847 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:30.183018 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.183246 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.183335 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:06:30.192327 1952 raft_consensus.cc:1261] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Refusing update from remote peer d03cb17ff3c24fd09dac021fcf624aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:30.192384 1673 raft_consensus.cc:1261] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Refusing update from remote peer d03cb17ff3c24fd09dac021fcf624aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:30.193348 2032 consensus_queue.cc:1035] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:30.193674 2029 consensus_queue.cc:1035] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:30.206019 2035 mvcc.cc:205] Tried to move back new op lower bound from 7333976433417629696 to 7333976433241473024. Current Snapshot: MvccSnapshot[applied={T|T < 7333976433417629696}]
I20260927 15:06:30.206584 2036 mvcc.cc:205] Tried to move back new op lower bound from 7333976433417629696 to 7333976433241473024. Current Snapshot: MvccSnapshot[applied={T|T < 7333976433417629696}]
I20260927 15:06:30.237072 1516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:30.237407 1516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:30.241503 1516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: Sending DeleteTablet for 3 replicas of tablet 272bdaa01d0a445eb6a8205705c92214
I20260927 15:06:30.243198 1653 tablet_service.cc:1514] Processing DeleteTablet for tablet 272bdaa01d0a445eb6a8205705c92214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34338
I20260927 15:06:30.243501 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 272bdaa01d0a445eb6a8205705c92214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34702
I20260927 15:06:30.244246 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 272bdaa01d0a445eb6a8205705c92214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59642
I20260927 15:06:30.244601 2046 tablet_replica.cc:330] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: stopping tablet replica
I20260927 15:06:30.245218 2046 raft_consensus.cc:2229] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:30.245867 2046 raft_consensus.cc:2258] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:30.246369 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:30.247442 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:30.255389 2047 tablet_replica.cc:330] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: stopping tablet replica
I20260927 15:06:30.256208 2047 raft_consensus.cc:2229] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:30.257388 2047 raft_consensus.cc:2258] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:30.257582 2046 ts_tablet_manager.cc:1905] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:30.257804 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 470cf03f1dff404f911a4606394186f1 (DEFAULT_TABLE table=test-table [id=399bd5a84d5f4d3bae95525d23e7c015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:30.258359 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470cf03f1dff404f911a4606394186f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.258833 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 470cf03f1dff404f911a4606394186f1 (DEFAULT_TABLE table=test-table [id=399bd5a84d5f4d3bae95525d23e7c015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:30.259384 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470cf03f1dff404f911a4606394186f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.259341 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 470cf03f1dff404f911a4606394186f1 (DEFAULT_TABLE table=test-table [id=399bd5a84d5f4d3bae95525d23e7c015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:30.259799 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470cf03f1dff404f911a4606394186f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.265554 2022 tablet_bootstrap.cc:492] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Bootstrap starting.
I20260927 15:06:30.267012 2021 tablet_bootstrap.cc:492] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Bootstrap starting.
I20260927 15:06:30.267290 2022 tablet_bootstrap.cc:654] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.268174 2046 ts_tablet_manager.cc:1918] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:30.268965 2047 ts_tablet_manager.cc:1905] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:30.268903 2021 tablet_bootstrap.cc:654] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.269411 2046 log.cc:1196] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/272bdaa01d0a445eb6a8205705c92214
I20260927 15:06:30.270144 2046 ts_tablet_manager.cc:1939] T 272bdaa01d0a445eb6a8205705c92214 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Deleting consensus metadata
I20260927 15:06:30.271998 2021 tablet_bootstrap.cc:492] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: No bootstrap required, opened a new log
I20260927 15:06:30.272169 2021 ts_tablet_manager.cc:1397] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 15:06:30.272287 1500 catalog_manager.cc:4841] TS 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3:40383): tablet 272bdaa01d0a445eb6a8205705c92214 (table test-table [id=78d2cb4f27a44ee390cc1d3a2182ca0d]) successfully deleted
I20260927 15:06:30.273247 2021 raft_consensus.cc:348] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.273546 2021 raft_consensus.cc:374] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.273664 2021 raft_consensus.cc:729] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf130632afc44e7a5a8fe6b0ff5e6ab, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.274181 2022 tablet_bootstrap.cc:492] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: No bootstrap required, opened a new log
I20260927 15:06:30.274091 2021 consensus_queue.cc:260] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.274354 2022 ts_tablet_manager.cc:1397] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 15:06:30.275345 2021 ts_tablet_manager.cc:1428] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 15:06:30.275462 2045 tablet_replica.cc:330] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: stopping tablet replica
I20260927 15:06:30.275830 2022 raft_consensus.cc:348] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.275990 2045 raft_consensus.cc:2229] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:30.276063 2022 raft_consensus.cc:374] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.276188 2022 raft_consensus.cc:729] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889b98be65044941b1904d940eb4956c, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.276544 2022 consensus_queue.cc:260] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.277110 2022 ts_tablet_manager.cc:1428] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 15:06:30.276549 2045 raft_consensus.cc:2258] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:30.284336 2023 tablet_bootstrap.cc:492] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Bootstrap starting.
I20260927 15:06:30.286142 2045 ts_tablet_manager.cc:1905] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:30.286968 2023 tablet_bootstrap.cc:654] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.290216 2047 ts_tablet_manager.cc:1918] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:30.290418 2047 log.cc:1196] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/272bdaa01d0a445eb6a8205705c92214
I20260927 15:06:30.291002 2047 ts_tablet_manager.cc:1939] T 272bdaa01d0a445eb6a8205705c92214 P d03cb17ff3c24fd09dac021fcf624aad: Deleting consensus metadata
I20260927 15:06:30.293081 1503 catalog_manager.cc:4841] TS d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903): tablet 272bdaa01d0a445eb6a8205705c92214 (table test-table [id=78d2cb4f27a44ee390cc1d3a2182ca0d]) successfully deleted
I20260927 15:06:30.293742 2045 ts_tablet_manager.cc:1918] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:30.293907 2045 log.cc:1196] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/272bdaa01d0a445eb6a8205705c92214
I20260927 15:06:30.294488 2045 ts_tablet_manager.cc:1939] T 272bdaa01d0a445eb6a8205705c92214 P 889b98be65044941b1904d940eb4956c: Deleting consensus metadata
I20260927 15:06:30.294905 2023 tablet_bootstrap.cc:492] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: No bootstrap required, opened a new log
I20260927 15:06:30.295105 2023 ts_tablet_manager.cc:1397] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260927 15:06:30.296005 2023 raft_consensus.cc:348] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.296250 2023 raft_consensus.cc:374] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.296303 1502 catalog_manager.cc:4841] TS 889b98be65044941b1904d940eb4956c (127.0.97.1:40201): tablet 272bdaa01d0a445eb6a8205705c92214 (table test-table [id=78d2cb4f27a44ee390cc1d3a2182ca0d]) successfully deleted
I20260927 15:06:30.296355 2023 raft_consensus.cc:729] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03cb17ff3c24fd09dac021fcf624aad, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.296671 2023 consensus_queue.cc:260] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.297315 2023 ts_tablet_manager.cc:1428] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 15:06:30.554327 2027 raft_consensus.cc:482] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:30.554607 2027 raft_consensus.cc:504] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.556728 2027 leader_election.cc:288] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889b98be65044941b1904d940eb4956c (127.0.97.1:40201), d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903)
I20260927 15:06:30.564339 2028 raft_consensus.cc:482] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:30.564657 2028 raft_consensus.cc:504] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.565843 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.557236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55487 (local address 127.0.97.1:40201)
0927 15:06:30.557437 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.557445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.558717 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 15:06:30.559044 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.559051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.559187 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.559263 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:30.560023 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.561176 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.563105 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.564438 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.564576 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.564693 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.565402 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.565551 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:06:30.566636 2028 leader_election.cc:288] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903), 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3:40383)
I20260927 15:06:30.567317 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.557316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40797 (local address 127.0.97.2:42903)
0927 15:06:30.557600 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.557606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.560486 (+ 2880us) server_negotiation.cc:407] Connection header received
0927 15:06:30.560657 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.560662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.560721 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.560819 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:30.561442 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.562602 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.564530 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.565800 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.565917 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.566012 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.566580 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.566742 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260927 15:06:30.567320 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.557044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40201 (local address 127.0.97.3:55487)
0927 15:06:30.558588 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.558616 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.558849 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.559475 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.559479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.559513 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:30.559814 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.559829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.561338 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.561344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.562833 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.562845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.564899 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.564905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.565190 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.565229 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.566271 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.566425 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":2654,"threads_started":1}
I20260927 15:06:30.568383 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470cf03f1dff404f911a4606394186f1" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889b98be65044941b1904d940eb4956c" is_pre_election: true
I20260927 15:06:30.568728 1673 raft_consensus.cc:2454] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 0.
I20260927 15:06:30.569447 1885 leader_election.cc:302] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889b98be65044941b1904d940eb4956c, 8bf130632afc44e7a5a8fe6b0ff5e6ab; no voters:
I20260927 15:06:30.570013 2027 raft_consensus.cc:2781] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:30.570152 2027 raft_consensus.cc:482] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:30.570266 2027 raft_consensus.cc:3037] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.574687 2027 raft_consensus.cc:504] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.576385 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470cf03f1dff404f911a4606394186f1" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889b98be65044941b1904d940eb4956c"
I20260927 15:06:30.576713 1673 raft_consensus.cc:3037] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.581835 1673 raft_consensus.cc:2454] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 1.
I20260927 15:06:30.582530 1885 leader_election.cc:302] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889b98be65044941b1904d940eb4956c, 8bf130632afc44e7a5a8fe6b0ff5e6ab; no voters:
I20260927 15:06:30.583513 2055 raft_consensus.cc:2781] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:30.583743 2027 leader_election.cc:288] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 election: Requested vote from peers 889b98be65044941b1904d940eb4956c (127.0.97.1:40201), d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903)
I20260927 15:06:30.585120 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.557038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42903 (local address 127.0.97.3:40797)
0927 15:06:30.560367 (+ 3329us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.560382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.560530 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.560845 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.560849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.560872 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:30.561185 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.561193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.562770 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.562776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.564328 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.564342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.566212 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.566221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.566385 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.566418 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.567573 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.584563 (+ 16990us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3224,"mutex_wait_us":385,"thread_start_us":2649,"threads_started":1}
I20260927 15:06:30.586326 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470cf03f1dff404f911a4606394186f1" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03cb17ff3c24fd09dac021fcf624aad" is_pre_election: true
I20260927 15:06:30.586908 1816 raft_consensus.cc:2454] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 0.
I20260927 15:06:30.588321 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.567990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44501 (local address 127.0.97.2:42903)
0927 15:06:30.568751 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.568757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.578653 (+ 9896us) server_negotiation.cc:407] Connection header received
0927 15:06:30.579596 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.579602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.579712 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.579784 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:30.580771 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.581916 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.585202 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.586726 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.586847 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.586976 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.587891 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.588069 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260927 15:06:30.586408 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470cf03f1dff404f911a4606394186f1" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03cb17ff3c24fd09dac021fcf624aad"
I20260927 15:06:30.588598 1814 raft_consensus.cc:3037] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.589308 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.567037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42903 (local address 127.0.97.1:44501)
0927 15:06:30.578079 (+ 11042us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.578103 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.579383 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.580010 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.580015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.580068 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:30.580563 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.580577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.582247 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.582252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.584933 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.584954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.587501 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.587507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.587700 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.587734 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.588743 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.588929 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10862,"thread_start_us":11889,"threads_started":1}
I20260927 15:06:30.589905 2055 raft_consensus.cc:686] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 LEADER]: Becoming Leader. State: Replica: 8bf130632afc44e7a5a8fe6b0ff5e6ab, State: Running, Role: LEADER
I20260927 15:06:30.590390 2055 consensus_queue.cc:237] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.596215 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470cf03f1dff404f911a4606394186f1" candidate_uuid: "889b98be65044941b1904d940eb4956c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03cb17ff3c24fd09dac021fcf624aad" is_pre_election: true
I20260927 15:06:30.602054 1814 raft_consensus.cc:2454] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 1.
I20260927 15:06:30.605139 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.568153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54221 (local address 127.0.97.3:40383)
0927 15:06:30.568313 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.568319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.587340 (+ 19021us) server_negotiation.cc:407] Connection header received
0927 15:06:30.587897 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.587905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.588004 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.588111 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:30.591443 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.592762 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.599324 (+ 6562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.601206 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.601388 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.601541 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.604044 (+ 2503us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.604276 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:06:30.605621 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.567540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40383 (local address 127.0.97.1:54221)
0927 15:06:30.587061 (+ 19521us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.587105 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.587257 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.588964 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.588971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.589005 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:30.589450 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.589463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.593025 (+ 3562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.593034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.598998 (+ 5964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.599026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.603180 (+ 4154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.603191 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.603457 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.603491 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.603755 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.604733 (+ 978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19365,"thread_start_us":22067,"threads_started":1}
I20260927 15:06:30.606494 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470cf03f1dff404f911a4606394186f1" candidate_uuid: "889b98be65044941b1904d940eb4956c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" is_pre_election: true
I20260927 15:06:30.607517 1603 leader_election.cc:302] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 889b98be65044941b1904d940eb4956c; no voters: 8bf130632afc44e7a5a8fe6b0ff5e6ab, d03cb17ff3c24fd09dac021fcf624aad
I20260927 15:06:30.608587 2028 raft_consensus.cc:2726] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 15:06:30.611689 1516 catalog_manager.cc:5495] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } health_report { overall_health: HEALTHY } } }
Sep 27 15:06:30 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:30.680357 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.665505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46729 (local address 127.0.0.1:44956)
0927 15:06:30.666223 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.666253 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.666442 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.666821 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.666827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.667589 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:30.667798 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.667808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.668953 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.668960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.669906 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.669919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.670615 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.670621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.670781 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.671557 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:30.671586 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:30.673918 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:30.677243 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:30.677263 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:30.677275 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:30.677951 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:30.678343 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:30.678348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:30.678352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:30.678554 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:30.679195 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:30.679204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:30.679405 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.679598 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.679947 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":356,"threads_started":1}
I20260927 15:06:30.680429 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.665582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.0.97.62:46729)
0927 15:06:30.666540 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.666547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.666577 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:06:30.666616 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.666619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.666677 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.666790 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:30.667958 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.668806 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.670055 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.670445 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.670589 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.674140 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:30.674146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:30.674149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:30.674177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:30.677049 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:30.678086 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:30.678090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:30.678093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:30.678231 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:30.678719 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:30.678724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:30.678730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:30.679018 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:30.679203 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.679715 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.680068 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":812,"thread_start_us":672,"threads_started":1}
I20260927 15:06:30.686007 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:06:30.687050 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:30.695103 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8be2a5e4324db3b1e9c350a94c5f1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cc2af914a6449c9a43aba42c2b024cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:06:30.695385 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8be2a5e4324db3b1e9c350a94c5f1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cc2af914a6449c9a43aba42c2b024cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:06:30.695425 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8be2a5e4324db3b1e9c350a94c5f1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cc2af914a6449c9a43aba42c2b024cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:06:30.695884 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8be2a5e4324db3b1e9c350a94c5f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.695907 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8be2a5e4324db3b1e9c350a94c5f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.695981 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8be2a5e4324db3b1e9c350a94c5f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.703986 2023 tablet_bootstrap.cc:492] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: Bootstrap starting.
I20260927 15:06:30.704844 2021 tablet_bootstrap.cc:492] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Bootstrap starting.
I20260927 15:06:30.705839 2023 tablet_bootstrap.cc:654] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.706663 2021 tablet_bootstrap.cc:654] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.710546 2023 tablet_bootstrap.cc:492] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: No bootstrap required, opened a new log
I20260927 15:06:30.710770 2023 ts_tablet_manager.cc:1397] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260927 15:06:30.711660 2023 raft_consensus.cc:348] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.712034 2023 raft_consensus.cc:374] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.712143 2023 raft_consensus.cc:729] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03cb17ff3c24fd09dac021fcf624aad, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.712494 2023 consensus_queue.cc:260] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.713054 2032 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.713433 2032 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.715901 2022 tablet_bootstrap.cc:492] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: Bootstrap starting.
I20260927 15:06:30.718904 2022 tablet_bootstrap.cc:654] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.723222 2021 tablet_bootstrap.cc:492] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: No bootstrap required, opened a new log
I20260927 15:06:30.723426 2021 ts_tablet_manager.cc:1397] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20260927 15:06:30.723990 2023 ts_tablet_manager.cc:1428] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260927 15:06:30.724316 2021 raft_consensus.cc:348] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.724702 2021 raft_consensus.cc:374] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.724787 2021 raft_consensus.cc:729] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf130632afc44e7a5a8fe6b0ff5e6ab, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.725072 2021 consensus_queue.cc:260] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.725723 2021 ts_tablet_manager.cc:1428] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:30.725677 2055 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.725972 2055 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.730615 2022 tablet_bootstrap.cc:492] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: No bootstrap required, opened a new log
I20260927 15:06:30.730823 2022 ts_tablet_manager.cc:1397] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20260927 15:06:30.731519 2022 raft_consensus.cc:348] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.731746 2022 raft_consensus.cc:374] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.731829 2022 raft_consensus.cc:729] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889b98be65044941b1904d940eb4956c, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.732084 2022 consensus_queue.cc:260] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.732697 2022 ts_tablet_manager.cc:1428] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:30.732546 2028 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.732838 2028 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.786356 2055 raft_consensus.cc:482] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:30.786589 2055 raft_consensus.cc:504] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.787297 2055 leader_election.cc:288] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889b98be65044941b1904d940eb4956c (127.0.97.1:40201), d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903)
I20260927 15:06:30.787850 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8be2a5e4324db3b1e9c350a94c5f1b" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03cb17ff3c24fd09dac021fcf624aad" is_pre_election: true
I20260927 15:06:30.787885 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8be2a5e4324db3b1e9c350a94c5f1b" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889b98be65044941b1904d940eb4956c" is_pre_election: true
I20260927 15:06:30.788129 1814 raft_consensus.cc:2454] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 0.
I20260927 15:06:30.788180 1673 raft_consensus.cc:2454] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 0.
I20260927 15:06:30.788615 1886 leader_election.cc:302] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf130632afc44e7a5a8fe6b0ff5e6ab, d03cb17ff3c24fd09dac021fcf624aad; no voters:
I20260927 15:06:30.788919 2055 raft_consensus.cc:2781] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:30.789009 2055 raft_consensus.cc:482] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:30.789127 2055 raft_consensus.cc:3037] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.791705 2055 raft_consensus.cc:504] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.792349 2055 leader_election.cc:288] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 election: Requested vote from peers 889b98be65044941b1904d940eb4956c (127.0.97.1:40201), d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903)
I20260927 15:06:30.792845 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8be2a5e4324db3b1e9c350a94c5f1b" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889b98be65044941b1904d940eb4956c"
I20260927 15:06:30.793102 1673 raft_consensus.cc:3037] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.793067 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8be2a5e4324db3b1e9c350a94c5f1b" candidate_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03cb17ff3c24fd09dac021fcf624aad"
I20260927 15:06:30.793303 1814 raft_consensus.cc:3037] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.795418 1814 raft_consensus.cc:2454] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 1.
I20260927 15:06:30.795440 1673 raft_consensus.cc:2454] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf130632afc44e7a5a8fe6b0ff5e6ab in term 1.
I20260927 15:06:30.795890 1886 leader_election.cc:302] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf130632afc44e7a5a8fe6b0ff5e6ab, d03cb17ff3c24fd09dac021fcf624aad; no voters:
I20260927 15:06:30.796370 2055 raft_consensus.cc:2781] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:30.796545 2055 raft_consensus.cc:686] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 LEADER]: Becoming Leader. State: Replica: 8bf130632afc44e7a5a8fe6b0ff5e6ab, State: Running, Role: LEADER
I20260927 15:06:30.796828 2055 consensus_queue.cc:237] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } }
I20260927 15:06:30.798416 2056 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: TxnStatusTablet state changed. Reason: New leader 8bf130632afc44e7a5a8fe6b0ff5e6ab. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.798827 2056 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:06:30.799471 2069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:06:30.799697 2069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:06:30.799988 1514 catalog_manager.cc:5495] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } health_report { overall_health: HEALTHY } } }
I20260927 15:06:30.826200 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.819747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.0.97.3:40383)
0927 15:06:30.819923 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.819932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.820020 (+ 88us) server_negotiation.cc:407] Connection header received
0927 15:06:30.820330 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.820334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.820408 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.820579 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:30.820952 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.821761 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.823243 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.824197 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.824338 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.824880 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:30.825488 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:30.825654 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.825754 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.825842 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:06:30.826200 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.819656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40383 (local address 127.0.0.1:34710)
0927 15:06:30.819917 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.819939 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.820106 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.820614 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.820619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.820630 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:06:30.820808 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.820817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.821918 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.821924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.823032 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.823044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.824371 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.824377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.824590 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.824663 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:06:30.825612 (+ 949us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:06:30.825638 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.825733 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.825895 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:06:30.830904 1673 raft_consensus.cc:1261] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Refusing update from remote peer 8bf130632afc44e7a5a8fe6b0ff5e6ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:30.830974 1814 raft_consensus.cc:1261] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Refusing update from remote peer 8bf130632afc44e7a5a8fe6b0ff5e6ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:30.831945 2056 consensus_queue.cc:1035] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:30.832417 2055 consensus_queue.cc:1035] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:30.840575 2032 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: TxnStatusTablet state changed. Reason: New leader 8bf130632afc44e7a5a8fe6b0ff5e6ab. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.841363 2032 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.844383 2056 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.844669 2056 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:06:30.852375 2032 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.852664 2032 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P d03cb17ff3c24fd09dac021fcf624aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.857167 2028 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: TxnStatusTablet state changed. Reason: New leader 8bf130632afc44e7a5a8fe6b0ff5e6ab. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.857664 2028 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.861068 2027 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.861406 2027 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 8bf130632afc44e7a5a8fe6b0ff5e6ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:06:30.868357 2028 tablet_replica.cc:439] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 } } peers { permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 } } peers { permanent_uuid: "8bf130632afc44e7a5a8fe6b0ff5e6ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40383 } } }
I20260927 15:06:30.868659 2028 tablet_replica.cc:442] T 6a8be2a5e4324db3b1e9c350a94c5f1b P 889b98be65044941b1904d940eb4956c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:30.869854 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.857147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.0.97.3:40383)
0927 15:06:30.857354 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.857364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.858693 (+ 1329us) server_negotiation.cc:407] Connection header received
0927 15:06:30.858987 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.858994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.859116 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.859201 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:30.863633 (+ 4432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.864808 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.867045 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.867824 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.867954 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.868714 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:30.868877 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:30.868976 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.869433 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.869540 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 15:06:30.890760 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.883095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36775 (local address 127.0.97.3:40383)
0927 15:06:30.883281 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.883288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.884133 (+ 845us) server_negotiation.cc:407] Connection header received
0927 15:06:30.884376 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.884384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.884480 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.884591 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:30.885763 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.886534 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.888826 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.889286 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.889398 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.889971 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:30.890105 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:30.890185 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.890407 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.890473 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":65}
I20260927 15:06:30.890933 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.883189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40383 (local address 127.0.97.3:36775)
0927 15:06:30.883974 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:30.884004 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:30.884195 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:30.884626 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:30.884638 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:30.884651 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:06:30.885556 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.885570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.886684 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.886691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.888624 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:30.888640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.889490 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.889497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:30.889683 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.889763 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:06:30.890210 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:06:30.890221 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:06:30.890308 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:06:30.890470 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":433,"threads_started":1}
I20260927 15:06:30.900949 1673 raft_consensus.cc:1261] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Refusing update from remote peer 8bf130632afc44e7a5a8fe6b0ff5e6ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:30.901010 1816 raft_consensus.cc:1261] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Refusing update from remote peer 8bf130632afc44e7a5a8fe6b0ff5e6ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:30.901525 2056 consensus_queue.cc:1035] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "889b98be65044941b1904d940eb4956c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:30.901930 2027 consensus_queue.cc:1035] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03cb17ff3c24fd09dac021fcf624aad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:30.989878 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:30.990166 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:30.993414 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ec75f43020a4be1925796540f74fa9a: Sending DeleteTablet for 3 replicas of tablet 470cf03f1dff404f911a4606394186f1
I20260927 15:06:30.994407 1653 tablet_service.cc:1514] Processing DeleteTablet for tablet 470cf03f1dff404f911a4606394186f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34338
I20260927 15:06:30.995232 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 470cf03f1dff404f911a4606394186f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59642
I20260927 15:06:30.995393 2089 tablet_replica.cc:330] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: stopping tablet replica
I20260927 15:06:30.995630 2089 raft_consensus.cc:2229] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:30.995960 2089 raft_consensus.cc:2258] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:30.995991 2090 tablet_replica.cc:330] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: stopping tablet replica
I20260927 15:06:30.996300 2090 raft_consensus.cc:2229] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:30.996681 2090 raft_consensus.cc:2258] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:30.997200 2089 ts_tablet_manager.cc:1905] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:30.997516 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 470cf03f1dff404f911a4606394186f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34702
I20260927 15:06:30.997857 2090 ts_tablet_manager.cc:1905] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:31.005371 2090 ts_tablet_manager.cc:1918] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:31.005528 2090 log.cc:1196] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/470cf03f1dff404f911a4606394186f1
I20260927 15:06:31.006067 2090 ts_tablet_manager.cc:1939] T 470cf03f1dff404f911a4606394186f1 P d03cb17ff3c24fd09dac021fcf624aad: Deleting consensus metadata
I20260927 15:06:31.006832 2089 ts_tablet_manager.cc:1918] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:31.007400 1503 catalog_manager.cc:4841] TS d03cb17ff3c24fd09dac021fcf624aad (127.0.97.2:42903): tablet 470cf03f1dff404f911a4606394186f1 (table test-table [id=399bd5a84d5f4d3bae95525d23e7c015]) successfully deleted
I20260927 15:06:31.007682 2089 log.cc:1196] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/470cf03f1dff404f911a4606394186f1
I20260927 15:06:31.008296 2089 ts_tablet_manager.cc:1939] T 470cf03f1dff404f911a4606394186f1 P 889b98be65044941b1904d940eb4956c: Deleting consensus metadata
I20260927 15:06:31.009693 1502 catalog_manager.cc:4841] TS 889b98be65044941b1904d940eb4956c (127.0.97.1:40201): tablet 470cf03f1dff404f911a4606394186f1 (table test-table [id=399bd5a84d5f4d3bae95525d23e7c015]) successfully deleted
I20260927 15:06:31.011677 2092 tablet_replica.cc:330] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: stopping tablet replica
I20260927 15:06:31.011969 2092 raft_consensus.cc:2229] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:31.012776 2092 raft_consensus.cc:2258] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:31.014372 2092 ts_tablet_manager.cc:1905] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:31.022794 2092 ts_tablet_manager.cc:1918] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:31.022956 2092 log.cc:1196] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/470cf03f1dff404f911a4606394186f1
I20260927 15:06:31.023411 2092 ts_tablet_manager.cc:1939] T 470cf03f1dff404f911a4606394186f1 P 8bf130632afc44e7a5a8fe6b0ff5e6ab: Deleting consensus metadata
I20260927 15:06:31.024488 1500 catalog_manager.cc:4841] TS 8bf130632afc44e7a5a8fe6b0ff5e6ab (127.0.97.3:40383): tablet 470cf03f1dff404f911a4606394186f1 (table test-table [id=399bd5a84d5f4d3bae95525d23e7c015]) successfully deleted
Sep 27 15:06:31 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:31.033957 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.014314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.0.97.1:40201)
0927 15:06:31.014506 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.014513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.015354 (+ 841us) server_negotiation.cc:407] Connection header received
0927 15:06:31.015555 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.015563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.015657 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.015773 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.017775 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.018985 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.020912 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.021544 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.021813 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:31.026792 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.026813 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.026834 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:31.026887 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.030752 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.031636 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.031648 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.031652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.031835 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.032203 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.032208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.032212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.032678 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.033001 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.033186 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.033562 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 15:06:31.035319 1718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34340
I20260927 15:06:31.053661 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.038506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.97.62:46729)
0927 15:06:31.038795 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.038800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.038942 (+ 142us) server_negotiation.cc:407] Connection header received
0927 15:06:31.039175 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.039179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.039241 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.039324 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.041071 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.041844 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.043736 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.044267 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.044369 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:31.047657 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.047663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.047666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:31.047710 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.050875 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.051803 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.051808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.051810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.051946 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.052354 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.052358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.052360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.052635 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.052827 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.053212 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.053389 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260927 15:06:31.054369 1514 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44962
I20260927 15:06:31.057557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1587
I20260927 15:06:31.071875 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.058492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.97.1:40201)
0927 15:06:31.058674 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.058701 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.058948 (+ 247us) server_negotiation.cc:407] Connection header received
0927 15:06:31.059202 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.059228 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.059318 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.059432 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.060583 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.061361 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.062611 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.063047 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.063262 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:31.065654 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.065669 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.065672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:31.065717 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.069095 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.069809 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.069816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.069820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.069956 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.070358 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.070365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.070368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.070633 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.070909 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.071251 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.071471 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 15:06:31.072494 1718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34350
W20260927 15:06:31.073124 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1587: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:31.073324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1587
I20260927 15:06:31.092120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1727
Sep 27 15:06:31 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:31.111532 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.092930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.0.97.2:42903)
0927 15:06:31.096966 (+ 4036us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.096971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.096990 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:06:31.097044 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.097048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.097116 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.097257 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.098466 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.099284 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.100676 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.101081 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.101229 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:31.105438 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.105452 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.105466 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:31.105505 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.108586 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.109330 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.109340 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.109344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.109487 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.109887 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.109893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.109896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.110276 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.110571 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.110901 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.111192 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3927,"thread_start_us":3820,"threads_started":1}
W20260927 15:06:31.112270 1858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59656
W20260927 15:06:31.112954 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:31.113133 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1727
I20260927 15:06:31.142711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1867
Sep 27 15:06:31 dist-test-slave-g0w1 krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:31.159495 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.143547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.0.97.3:40383)
0927 15:06:31.143751 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.143759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.143781 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:06:31.143979 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.143984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.144084 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.144201 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.145493 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.146279 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.147706 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:31.148187 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:31.148355 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:31.152889 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.152901 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.152915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:31.152961 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.156383 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.157185 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.157193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.157196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.157345 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:31.157803 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:31.157809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:31.157812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:31.158225 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.158563 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.158899 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.159182 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 15:06:31.160233 1997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34714
W20260927 15:06:31.161109 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1867: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:31.161319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1867
I20260927 15:06:31.180912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 1484
I20260927 15:06:31.182219 1556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:31.297503 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 1484
2026-09-27T15:06:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5077 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:31 dist-test-slave-g0w1 krb5kdc[2101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:31 dist-test-slave-g0w1 krb5kdc[2101](info): set up 2 sockets
Sep 27 15:06:31 dist-test-slave-g0w1 krb5kdc[2101](info): commencing operation
krb5kdc: starting...
W20260927 15:06:32.403458 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:32 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:32.598647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34015
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:32.938207 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:32.938519 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:32.938695 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:32.946099 2114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:32.946220 2114 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:06:32.946308 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:32.946434 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:32.946527 2114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:32.946640 2114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:32.956970 2114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34015
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:32.958335 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:32.959812 2114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:32.972903 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:32.973084 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:32.973106 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:32.974280 2114 server_base.cc:1034] running on GCE node
I20260927 15:06:32.975261 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:32.976861 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:32.978097 2114 hybrid_clock.cc:648] HybridClock initialized: now 1790521592978032 us; error 37 us; skew 500 ppm
I20260927 15:06:32.978431 2114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:32 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:32.982165 2114 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:32.983855 2114 webserver.cc:460] Webserver started at http://127.0.97.62:36559/ using document root <none> and password file <none>
I20260927 15:06:32.984611 2114 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:32.984700 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:32.985050 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:32.988245 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "77db2be6617c4b529674c3f5f70c2f32"
format_stamp: "Formatted at 2026-09-27 15:06:32 on dist-test-slave-g0w1"
server_key: "414292252fad18a64c4d672e0d15aba1"
server_key_iv: "9593d8a9ae7b85e2a42f42d988f9c143"
server_key_version: "encryptionkey@0"
I20260927 15:06:32.989094 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "77db2be6617c4b529674c3f5f70c2f32"
format_stamp: "Formatted at 2026-09-27 15:06:32 on dist-test-slave-g0w1"
server_key: "414292252fad18a64c4d672e0d15aba1"
server_key_iv: "9593d8a9ae7b85e2a42f42d988f9c143"
server_key_version: "encryptionkey@0"
I20260927 15:06:32.994563 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:32.998298 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:32.999929 2114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:33.000130 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "77db2be6617c4b529674c3f5f70c2f32"
format_stamp: "Formatted at 2026-09-27 15:06:32 on dist-test-slave-g0w1"
server_key: "414292252fad18a64c4d672e0d15aba1"
server_key_iv: "9593d8a9ae7b85e2a42f42d988f9c143"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.000383 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:33.021947 2114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:33.026443 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:33.026749 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:33.049808 2114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34015
I20260927 15:06:33.049853 2190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34015 every 8 connection(s)
I20260927 15:06:33.051301 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:33.055181 2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:33.057991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 2114
I20260927 15:06:33.058198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:33.058986 388 external_mini_cluster.cc:1428] Setting key 6b68b80f0587328c66674d04273f818b
I20260927 15:06:33.068501 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Bootstrap starting.
I20260927 15:06:33.072693 2191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:33.074441 2191 log.cc:824] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:06:33 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:33.079633 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: No bootstrap required, opened a new log
I20260927 15:06:33.086688 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.061376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.97.62:34015)
0927 15:06:33.062231 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:33.062243 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:33.062288 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:06:33.064117 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:33.064143 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:33.064605 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:33.064995 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:33.066868 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.068338 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.070145 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.070930 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.071210 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.076796 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:33.076809 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:33.076833 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:33.076881 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:33.081102 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.082164 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.082173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.082184 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.082414 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.082891 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.082898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.082907 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.083778 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:33.084073 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:33.084632 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 15:06:33.085112 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":421,"threads_started":1}
I20260927 15:06:33.090122 2191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db2be6617c4b529674c3f5f70c2f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34015 } }
I20260927 15:06:33.090597 2191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:33.090667 2191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77db2be6617c4b529674c3f5f70c2f32, State: Initialized, Role: FOLLOWER
I20260927 15:06:33.091310 2191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db2be6617c4b529674c3f5f70c2f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34015 } }
I20260927 15:06:33.091526 2191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:33.091622 2191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:33.091748 2191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:33.122182 2191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db2be6617c4b529674c3f5f70c2f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34015 } }
I20260927 15:06:33.123112 2191 leader_election.cc:302] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77db2be6617c4b529674c3f5f70c2f32; no voters:
I20260927 15:06:33.123942 2191 leader_election.cc:288] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:33.124055 2196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:33.124888 2196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [term 1 LEADER]: Becoming Leader. State: Replica: 77db2be6617c4b529674c3f5f70c2f32, State: Running, Role: LEADER
I20260927 15:06:33.125589 2196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db2be6617c4b529674c3f5f70c2f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34015 } }
I20260927 15:06:33.126844 2191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:33.128803 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77db2be6617c4b529674c3f5f70c2f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db2be6617c4b529674c3f5f70c2f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34015 } } }
I20260927 15:06:33.129297 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:33.128791 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77db2be6617c4b529674c3f5f70c2f32. Latest consensus state: current_term: 1 leader_uuid: "77db2be6617c4b529674c3f5f70c2f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77db2be6617c4b529674c3f5f70c2f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34015 } } }
I20260927 15:06:33.129465 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:33.131095 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:33.137248 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:33.146695 2201 catalog_manager.cc:1347] Generated new cluster ID: b9b5e9a4ea0849d7922059fffc438f66
I20260927 15:06:33.146869 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:33.186359 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:33.187811 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:33.196892 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Generated new TSK 0
I20260927 15:06:33.198002 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:33.247517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34015
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:33.588442 2217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:33.588798 2217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:33.588972 2217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:33.596917 2217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:33.597054 2217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:33.597210 2217 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:06:33.597295 2217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:06:33.597425 2217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:33.607887 2217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:33.609294 2217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:33.610699 2217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:33.624668 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:33.624795 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:33.624645 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:33.625502 2217 server_base.cc:1034] running on GCE node
I20260927 15:06:33.626246 2217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:33.627338 2217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:33.628726 2217 hybrid_clock.cc:648] HybridClock initialized: now 1790521593628650 us; error 60 us; skew 500 ppm
I20260927 15:06:33.629091 2217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:33 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:33.636780 2217 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:33.638484 2217 webserver.cc:460] Webserver started at http://127.0.97.1:36949/ using document root <none> and password file <none>
I20260927 15:06:33.639429 2217 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:33.639597 2217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:33.639938 2217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:33.643309 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "b749d353c10741e6bdde4b8a9a58d49b"
format_stamp: "Formatted at 2026-09-27 15:06:33 on dist-test-slave-g0w1"
server_key: "bb3f80b26591df220f721633ff5f758b"
server_key_iv: "b0fff6ba2e33f8ded594dc22909e8273"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.644199 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b749d353c10741e6bdde4b8a9a58d49b"
format_stamp: "Formatted at 2026-09-27 15:06:33 on dist-test-slave-g0w1"
server_key: "bb3f80b26591df220f721633ff5f758b"
server_key_iv: "b0fff6ba2e33f8ded594dc22909e8273"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.649874 2217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 15:06:33.653795 2232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:33.655406 2217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:33.655599 2217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "b749d353c10741e6bdde4b8a9a58d49b"
format_stamp: "Formatted at 2026-09-27 15:06:33 on dist-test-slave-g0w1"
server_key: "bb3f80b26591df220f721633ff5f758b"
server_key_iv: "b0fff6ba2e33f8ded594dc22909e8273"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.655855 2217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:33.692873 2217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:33.697522 2217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:33.697847 2217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:33.700779 2217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:33.700955 2217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:33.701097 2217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:33.701186 2217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:06:33 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:33.726227 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.701826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53061 (local address 127.0.97.62:34015)
0927 15:06:33.702387 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:33.702396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:33.704068 (+ 1672us) server_negotiation.cc:407] Connection header received
0927 15:06:33.706852 (+ 2784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:33.706862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:33.706987 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:33.707171 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:33.709757 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.711086 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.713652 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.714084 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.714233 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.719522 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:33.719528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:33.719531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:33.719567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:33.722452 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.723423 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.723431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.723434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.723636 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.724167 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.724172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.724175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.724451 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:33.724615 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:33.725575 (+ 960us) server_negotiation.cc:299] Negotiation successful
0927 15:06:33.725817 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":318,"threads_started":1}
I20260927 15:06:33.728101 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.702677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.97.1:53061)
0927 15:06:33.703881 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:33.703949 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:33.706420 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:33.707471 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:33.707485 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:33.708414 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:33.709493 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:33.709523 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.711343 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.711349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:33.713447 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:33.713462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.714358 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.714365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:33.714765 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.715774 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:33.715798 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:33.719246 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:33.722648 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:33.722658 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:33.722679 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:33.723264 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:33.723792 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:33.723798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:33.723806 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:33.724021 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:33.724588 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:33.724598 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:33.725167 (+ 569us) client_negotiation.cc:769] Sending connection context
0927 15:06:33.725473 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 15:06:33.726061 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":460,"threads_started":1}
I20260927 15:06:33.757227 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.741469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48935 (local address 127.0.97.62:34015)
0927 15:06:33.741702 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:33.741712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:33.742404 (+ 692us) server_negotiation.cc:407] Connection header received
0927 15:06:33.742649 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:33.742653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:33.742719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:33.742811 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:33.744582 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.745610 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.747197 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.747620 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.747779 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.750022 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:33.750027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:33.750030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:33.750064 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:33.753788 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.754795 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.754804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.754810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.754983 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.755521 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.755528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.755531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.755980 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:33.756131 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:33.756598 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 15:06:33.756841 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:06:33.757427 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.741446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.97.1:48935)
0927 15:06:33.742267 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:33.742294 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:33.742490 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:33.742881 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:33.742889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:33.743754 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:33.744362 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:33.744381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.745811 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.745816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:33.746955 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:33.746970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.747822 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.747828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:33.748036 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.748964 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:33.748984 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:33.749792 (+ 808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:33.753990 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:33.754003 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:33.754010 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:33.754634 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:33.755153 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:33.755162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:33.755168 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:33.755370 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:33.756096 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:33.756100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:33.756316 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 15:06:33.756487 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:06:33.756906 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":407,"threads_started":1}
I20260927 15:06:33.760335 2217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35721
I20260927 15:06:33.760347 2352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35721 every 8 connection(s)
I20260927 15:06:33.761751 2217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:33.764249 2353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34015
I20260927 15:06:33.764464 2353 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:33.765180 2353 heartbeater.cc:511] Master 127.0.97.62:34015 requested a full tablet report, sending...
I20260927 15:06:33.767227 2146 ts_manager.cc:194] Registered new tserver with Master: b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721)
I20260927 15:06:33.768855 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53061
I20260927 15:06:33.768973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 2217
I20260927 15:06:33.769181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:33.769930 388 external_mini_cluster.cc:1428] Setting key 9115aa984fbbf50825583c19d5755fa1
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:33.815178 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34015
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:34.183403 2357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:34.183706 2357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:34.183879 2357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:34.191459 2357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:34.191589 2357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:34.191738 2357 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:06:34.191799 2357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:06:34.191952 2357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:34.202288 2357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:34.203832 2357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:34.205317 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:34.219775 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:34.219691 2363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:34.219908 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:34.220798 2357 server_base.cc:1034] running on GCE node
I20260927 15:06:34.221462 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:34.222553 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:34.223825 2357 hybrid_clock.cc:648] HybridClock initialized: now 1790521594223769 us; error 48 us; skew 500 ppm
I20260927 15:06:34.224174 2357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:34 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:34.227818 2357 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:34.229643 2357 webserver.cc:460] Webserver started at http://127.0.97.2:36001/ using document root <none> and password file <none>
I20260927 15:06:34.230433 2357 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:34.230612 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:34.230960 2357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:34.234282 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "12388ef6606048d4900d86655b8babe3"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-g0w1"
server_key: "6b5fa986616cb45b02fc1ffd446a8bd4"
server_key_iv: "f2045f17f808ac8b4f969162c3dd4aeb"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.235262 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "12388ef6606048d4900d86655b8babe3"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-g0w1"
server_key: "6b5fa986616cb45b02fc1ffd446a8bd4"
server_key_iv: "f2045f17f808ac8b4f969162c3dd4aeb"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.241178 2357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:06:34.245220 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:34.246853 2357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:06:34.247115 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "12388ef6606048d4900d86655b8babe3"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-g0w1"
server_key: "6b5fa986616cb45b02fc1ffd446a8bd4"
server_key_iv: "f2045f17f808ac8b4f969162c3dd4aeb"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.247376 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:34.275022 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:34.280409 2357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:34.280750 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:34.283675 2357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:34.283835 2357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:34.283998 2357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:34.284106 2357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:06:34 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:34.319442 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.284601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53785 (local address 127.0.97.62:34015)
0927 15:06:34.299797 (+ 15196us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.299803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.299828 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:34.299924 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.299932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.300042 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.300408 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.302320 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.303182 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.306012 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.306463 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.306637 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.311431 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.311438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.311442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:34.311487 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.315038 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.316518 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.316524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.316528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.316726 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.317457 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.317463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.317466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.317788 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.317949 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.318769 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.319015 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15073,"thread_start_us":15609,"threads_started":1}
I20260927 15:06:34.321112 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.285564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.97.2:53785)
0927 15:06:34.286826 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.286891 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.290092 (+ 3201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.300748 (+ 10656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.300760 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.301403 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:34.302089 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.302109 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.303367 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.303379 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.305838 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.305855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.306847 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.306859 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.307187 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.307985 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:34.308009 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:34.311039 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:34.315296 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.315312 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.315336 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.316287 (+ 951us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.316897 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.316909 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.316916 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.317297 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.317946 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:34.317956 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:34.318364 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.318658 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.319301 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":551,"threads_started":1}
I20260927 15:06:34.345762 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.328621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43919 (local address 127.0.97.62:34015)
0927 15:06:34.328851 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.328856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.329522 (+ 666us) server_negotiation.cc:407] Connection header received
0927 15:06:34.329803 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.329807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.329888 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.330002 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.331135 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.332062 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.334411 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.334851 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.335010 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.337572 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.337588 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.337597 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:34.337739 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.342204 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.343306 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.343315 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.343320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.343557 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.344145 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.344149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.344151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.344522 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.344771 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.345281 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.345481 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:06:34.346400 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.328671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.97.2:43919)
0927 15:06:34.329396 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.329415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.329606 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.330032 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.330037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.330610 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:34.330930 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.330941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.332269 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.332274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.334226 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.334242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.335058 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.335064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.335340 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.336095 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:34.336114 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:34.337138 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:34.342388 (+ 5250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.342394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.342397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.343142 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.343746 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.343756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.343763 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.343987 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.344687 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:34.344699 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:34.344983 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.345196 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.345716 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":394,"threads_started":1}
I20260927 15:06:34.350055 2357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33249
I20260927 15:06:34.350085 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33249 every 8 connection(s)
I20260927 15:06:34.351725 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:34.354061 2493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34015
I20260927 15:06:34.354314 2493 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:34.354969 2493 heartbeater.cc:511] Master 127.0.97.62:34015 requested a full tablet report, sending...
I20260927 15:06:34.356451 2146 ts_manager.cc:194] Registered new tserver with Master: 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249)
I20260927 15:06:34.357285 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53785
I20260927 15:06:34.357820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 2357
I20260927 15:06:34.357995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:34.358529 388 external_mini_cluster.cc:1428] Setting key 417583ac4b469e7128d635d76e40a1fe
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:34.404179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34015
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:34.742178 2497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:34.742473 2497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:34.742653 2497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:34.750125 2497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:34.750257 2497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:34.750407 2497 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:06:34.750494 2497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:06:34.750626 2497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:34.762851 2497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:34.764448 2497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:34.765753 2497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:06:34.772931 2353 heartbeater.cc:503] Master 127.0.97.62:34015 was elected leader, sending a full tablet report...
W20260927 15:06:34.780526 2502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:34.780604 2503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:34.780629 2505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:34.781598 2497 server_base.cc:1034] running on GCE node
I20260927 15:06:34.782371 2497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:34.783535 2497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:34.784803 2497 hybrid_clock.cc:648] HybridClock initialized: now 1790521594784732 us; error 69 us; skew 500 ppm
I20260927 15:06:34.785180 2497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:34 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:34.788702 2497 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:34.790277 2497 webserver.cc:460] Webserver started at http://127.0.97.3:41833/ using document root <none> and password file <none>
I20260927 15:06:34.790994 2497 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:34.791180 2497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:34.791473 2497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:34.794521 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "bfb9e58b54744290bef288ece3234b7e"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-g0w1"
server_key: "20780f695571bdd7c0b8e77297cf58f9"
server_key_iv: "1970619521ae241e2ea5efb663bc50ba"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.795394 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bfb9e58b54744290bef288ece3234b7e"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-g0w1"
server_key: "20780f695571bdd7c0b8e77297cf58f9"
server_key_iv: "1970619521ae241e2ea5efb663bc50ba"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.800999 2497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:06:34.805853 2512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:34.807816 2497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:06:34.807988 2497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "bfb9e58b54744290bef288ece3234b7e"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-g0w1"
server_key: "20780f695571bdd7c0b8e77297cf58f9"
server_key_iv: "1970619521ae241e2ea5efb663bc50ba"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.808212 2497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:34.835819 2497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:34.840714 2497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:34.841053 2497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:34.844100 2497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:34.844204 2497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:34.844372 2497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:34.844463 2497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:06:34 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:34.874262 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.844872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45325 (local address 127.0.97.62:34015)
0927 15:06:34.845210 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.845216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.846750 (+ 1534us) server_negotiation.cc:407] Connection header received
0927 15:06:34.850044 (+ 3294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.850052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.850161 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.850244 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.853429 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.854769 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.858236 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.859035 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.859274 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.865959 (+ 6685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.865966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.865973 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:34.866026 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.869137 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.870533 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.870543 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.870549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.870742 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.871540 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.871548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.871554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.871962 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.872205 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.873616 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.873929 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260927 15:06:34.877650 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.845556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.97.3:45325)
0927 15:06:34.846510 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.846568 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.849605 (+ 3037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.850925 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.850935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.852052 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:34.853154 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.853191 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.855001 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.855010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.857960 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.857987 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.859443 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.859457 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.859940 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.861074 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:34.861125 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:34.865559 (+ 4434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:34.869342 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.869355 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.869384 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.870312 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.870935 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.870943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.870948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.871361 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.872324 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:34.872338 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:34.873016 (+ 678us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.874605 (+ 1589us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.875460 (+ 855us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":405,"threads_started":1}
I20260927 15:06:34.890722 2497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40831
I20260927 15:06:34.890827 2631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40831 every 8 connection(s)
I20260927 15:06:34.892161 2497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:34.895511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 2497
I20260927 15:06:34.895625 2632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34015
I20260927 15:06:34.895704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:34.895895 2632 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:34.896180 388 external_mini_cluster.cc:1428] Setting key 0a5225437f5b97fdea92cd58bde572d3
I20260927 15:06:34.896782 2632 heartbeater.cc:511] Master 127.0.97.62:34015 requested a full tablet report, sending...
I20260927 15:06:34.898428 2146 ts_manager.cc:194] Registered new tserver with Master: bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831)
I20260927 15:06:34.899610 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:34.899648 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45325
I20260927 15:06:34.901194 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.885407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49121 (local address 127.0.97.62:34015)
0927 15:06:34.885578 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.885583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.886340 (+ 757us) server_negotiation.cc:407] Connection header received
0927 15:06:34.886595 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.886602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.886682 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.886848 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.888458 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.889391 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.891528 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.891972 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.892121 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.894569 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.894574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.894577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:34.894614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.898192 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.898999 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.899004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.899007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.899267 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.899669 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.899742 (+ 73us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.899746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.900020 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.900193 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.900591 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.900817 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:06:34.901335 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.885497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.97.3:49121)
0927 15:06:34.886176 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.886193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.886429 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.886878 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.886883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.887712 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:34.888272 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.888290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.889558 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.889564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.891364 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.891378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.892180 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.892188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.892375 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.893070 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:34.893100 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:34.894379 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:34.898342 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.898348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.898352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.898861 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.899403 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:34.899408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:34.899412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:34.899547 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:34.900159 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:34.900165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:34.900357 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.900505 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.900853 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":413,"threads_started":1}
Sep 27 15:06:34 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:06:34 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:34.951335 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.936128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.0.97.62:34015)
0927 15:06:34.936330 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.936335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.936635 (+ 300us) server_negotiation.cc:407] Connection header received
0927 15:06:34.936868 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.936873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.936953 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.937063 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.938591 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.939542 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.940878 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.941296 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.941407 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.945560 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.945566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.945569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:34.945612 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.948486 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.949256 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.949262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.949267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.949403 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:34.949812 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:34.949817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:34.949820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:34.950110 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.950369 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.950708 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.950971 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:06:34.955745 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:34.962864 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:34.999784 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.977106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40831 (local address 127.0.0.1:50702)
0927 15:06:34.991639 (+ 14533us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.991658 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.991884 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.992546 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.992552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.992603 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:34.993075 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.993085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.994545 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.994550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.996869 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.996885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.998625 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.998631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.998844 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.998899 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.999033 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.999288 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14399,"mutex_wait_us":88,"spinlock_wait_cycles":4992,"thread_start_us":9694,"threads_started":1}
I20260927 15:06:34.999949 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.977674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.0.97.3:40831)
0927 15:06:34.991015 (+ 13341us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.991028 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.991782 (+ 754us) server_negotiation.cc:407] Connection header received
0927 15:06:34.992080 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.992090 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.992336 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.992536 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:34.993281 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.994390 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.997039 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.998419 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.999138 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.999364 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.999433 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.999600 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13138,"thread_start_us":12948,"threads_started":1}
I20260927 15:06:35.000738 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.976674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.97.2:33249)
0927 15:06:34.991353 (+ 14679us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.991365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.991402 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:06:34.991518 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.991527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.991859 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.992079 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:34.992953 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.994665 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.997509 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.998832 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.998963 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.999223 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.000287 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.000457 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14481,"thread_start_us":14335,"threads_started":1}
I20260927 15:06:35.001122 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.976570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33249 (local address 127.0.0.1:47686)
0927 15:06:34.987420 (+ 10850us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.987490 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.987790 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.992086 (+ 4296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.992093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.992150 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:34.992741 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.992759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.994832 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.994840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.997328 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.997344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.999378 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.999386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.999548 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.999580 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.999672 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.999825 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10428,"mutex_wait_us":28,"spinlock_wait_cycles":14080,"thread_start_us":15011,"threads_started":1}
I20260927 15:06:35.001222 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.976836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35721 (local address 127.0.0.1:36118)
0927 15:06:34.990090 (+ 13254us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:34.990116 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:34.990267 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:34.992278 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:34.992284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:34.992327 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:34.993002 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.993021 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.995117 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.995127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.997710 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:34.997719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.999237 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.999243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:34.999442 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.999477 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:06:34.999543 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:06:34.999673 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13070,"mutex_wait_us":7,"spinlock_wait_cycles":768,"thread_start_us":12879,"threads_started":1}
I20260927 15:06:35.001588 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.977520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.0.97.1:35721)
0927 15:06:34.991663 (+ 14143us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.991671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.991699 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:06:34.991811 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.991816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.992048 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.993230 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:34.993458 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.994928 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.997860 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.998994 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.000283 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.000490 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.000559 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.000714 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13985,"thread_start_us":13850,"threads_started":1}
I20260927 15:06:35.003895 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbe61dc2f7a46228982e9447d3201ec (DEFAULT_TABLE table=test-table [id=b527130737ab45c3a7fbb38971e92126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:35.004761 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbe61dc2f7a46228982e9447d3201ec (DEFAULT_TABLE table=test-table [id=b527130737ab45c3a7fbb38971e92126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:35.004976 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbe61dc2f7a46228982e9447d3201ec (DEFAULT_TABLE table=test-table [id=b527130737ab45c3a7fbb38971e92126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:35.005560 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbe61dc2f7a46228982e9447d3201ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.006094 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbe61dc2f7a46228982e9447d3201ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.006096 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbe61dc2f7a46228982e9447d3201ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.018379 2651 tablet_bootstrap.cc:492] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Bootstrap starting.
I20260927 15:06:35.020934 2653 tablet_bootstrap.cc:492] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Bootstrap starting.
I20260927 15:06:35.022882 2651 tablet_bootstrap.cc:654] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.023639 2652 tablet_bootstrap.cc:492] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Bootstrap starting.
I20260927 15:06:35.024991 2651 log.cc:824] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:35.027455 2652 tablet_bootstrap.cc:654] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.027453 2653 tablet_bootstrap.cc:654] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.028226 2651 tablet_bootstrap.cc:492] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: No bootstrap required, opened a new log
I20260927 15:06:35.028504 2651 ts_tablet_manager.cc:1397] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 15:06:35.028795 2652 log.cc:824] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:35.028805 2653 log.cc:824] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:35.037891 2653 tablet_bootstrap.cc:492] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: No bootstrap required, opened a new log
I20260927 15:06:35.037781 2651 raft_consensus.cc:348] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.038249 2652 tablet_bootstrap.cc:492] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: No bootstrap required, opened a new log
I20260927 15:06:35.038235 2653 ts_tablet_manager.cc:1397] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260927 15:06:35.038301 2651 raft_consensus.cc:374] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.038391 2651 raft_consensus.cc:729] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.038538 2652 ts_tablet_manager.cc:1397] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260927 15:06:35.039165 2651 consensus_queue.cc:260] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.040392 2632 heartbeater.cc:503] Master 127.0.97.62:34015 was elected leader, sending a full tablet report...
I20260927 15:06:35.041188 2651 ts_tablet_manager.cc:1428] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 15:06:35.047950 2653 raft_consensus.cc:348] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.048396 2653 raft_consensus.cc:374] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.048517 2653 raft_consensus.cc:729] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12388ef6606048d4900d86655b8babe3, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.048270 2652 raft_consensus.cc:348] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.048744 2652 raft_consensus.cc:374] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.048856 2652 raft_consensus.cc:729] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.049247 2653 consensus_queue.cc:260] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.049479 2652 consensus_queue.cc:260] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.050544 2493 heartbeater.cc:503] Master 127.0.97.62:34015 was elected leader, sending a full tablet report...
I20260927 15:06:35.052575 2653 ts_tablet_manager.cc:1428] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260927 15:06:35.054323 2652 ts_tablet_manager.cc:1428] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
W20260927 15:06:35.104043 2494 tablet.cc:2369] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:35.142666 2657 raft_consensus.cc:482] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:35.143015 2657 raft_consensus.cc:504] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
W20260927 15:06:35.144479 2633 tablet.cc:2369] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:35.145516 2657 leader_election.cc:288] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249), b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721)
I20260927 15:06:35.151521 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.145199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50831 (local address 127.0.97.1:35721)
0927 15:06:35.145404 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.145411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.145628 (+ 217us) server_negotiation.cc:407] Connection header received
0927 15:06:35.145834 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.145838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.145899 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.145986 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:35.146532 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.147532 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.149678 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.150636 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.150788 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.150901 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.151127 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.151262 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:06:35.151554 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.145141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35721 (local address 127.0.97.3:50831)
0927 15:06:35.145493 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.145517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.145686 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.146051 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.146059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.146102 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:35.146364 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.146378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.147694 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.147698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.149509 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.149524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.150811 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.150816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.150952 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.150976 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.151052 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.151212 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260927 15:06:35.152746 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.145135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60153 (local address 127.0.97.2:33249)
0927 15:06:35.145383 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.145390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.146611 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 15:06:35.146767 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.146772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.146925 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.147044 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:35.147621 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.148543 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.150466 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.151792 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.151922 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.152029 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.152360 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.152501 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 15:06:35.152763 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.145711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33249 (local address 127.0.97.3:60153)
0927 15:06:35.146479 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.146497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.146625 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.147062 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.147089 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.147113 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:35.147451 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.147462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.148697 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.148704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.150319 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.150330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.151995 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.152006 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.152186 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.152216 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.152317 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.152460 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":454,"threads_started":1}
I20260927 15:06:35.152838 2307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe61dc2f7a46228982e9447d3201ec" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b749d353c10741e6bdde4b8a9a58d49b" is_pre_election: true
I20260927 15:06:35.153442 2307 raft_consensus.cc:2454] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 0.
I20260927 15:06:35.153770 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe61dc2f7a46228982e9447d3201ec" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3" is_pre_election: true
I20260927 15:06:35.154321 2443 raft_consensus.cc:2454] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 0.
I20260927 15:06:35.154304 2515 leader_election.cc:302] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:35.154767 2657 raft_consensus.cc:2781] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:35.154953 2657 raft_consensus.cc:482] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:35.155126 2657 raft_consensus.cc:3037] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.157375 2657 raft_consensus.cc:504] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.158015 2657 leader_election.cc:288] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 election: Requested vote from peers 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249), b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721)
I20260927 15:06:35.158612 2307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe61dc2f7a46228982e9447d3201ec" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b749d353c10741e6bdde4b8a9a58d49b"
I20260927 15:06:35.158669 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe61dc2f7a46228982e9447d3201ec" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3"
I20260927 15:06:35.158844 2307 raft_consensus.cc:3037] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.158860 2443 raft_consensus.cc:3037] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.162145 2443 raft_consensus.cc:2454] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 1.
I20260927 15:06:35.162147 2307 raft_consensus.cc:2454] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 1.
I20260927 15:06:35.162662 2515 leader_election.cc:302] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:35.163045 2657 raft_consensus.cc:2781] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:35.163540 2657 raft_consensus.cc:686] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 1 LEADER]: Becoming Leader. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Running, Role: LEADER
I20260927 15:06:35.164297 2657 consensus_queue.cc:237] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.169171 2145 catalog_manager.cc:5495] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb9e58b54744290bef288ece3234b7e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bfb9e58b54744290bef288ece3234b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } health_report { overall_health: HEALTHY } } }
Sep 27 15:06:35 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:35.223685 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.209666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.0.97.62:34015)
0927 15:06:35.209870 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.209877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.210424 (+ 547us) server_negotiation.cc:407] Connection header received
0927 15:06:35.210640 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.210645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.210752 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.210859 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:35.212108 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.212930 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.214208 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.214607 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.214754 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.218121 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:35.218126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:35.218129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:35.218159 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:35.220832 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:35.221558 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:35.221566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:35.221569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:35.221683 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:35.222045 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:35.222051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:35.222054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:35.222343 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:35.222518 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.222925 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.223333 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":212,"server-negotiator.queue_time_us":92}
I20260927 15:06:35.223717 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.209728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34015 (local address 127.0.0.1:41634)
0927 15:06:35.210311 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.210327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.210516 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.210882 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.210885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.211680 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:35.211949 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.211959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.213063 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.213070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.214061 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.214072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.214791 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.214796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.214954 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.215730 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:35.215749 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:35.217917 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:35.220994 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:35.221009 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:35.221017 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:35.221436 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:35.221789 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:35.221795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:35.221799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:35.221934 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:35.222469 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:35.222480 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:35.222656 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.222820 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.223210 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":320,"threads_started":1}
I20260927 15:06:35.229600 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41634:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:06:35.230706 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:35.238286 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7bbf729c6f48729297c0d2efa9e4d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d42694f119f4eeab846983d87467a5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:06:35.238176 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7bbf729c6f48729297c0d2efa9e4d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d42694f119f4eeab846983d87467a5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:06:35.238176 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7bbf729c6f48729297c0d2efa9e4d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d42694f119f4eeab846983d87467a5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:06:35.239151 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7bbf729c6f48729297c0d2efa9e4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.239161 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7bbf729c6f48729297c0d2efa9e4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.239161 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7bbf729c6f48729297c0d2efa9e4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.244946 2651 tablet_bootstrap.cc:492] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: Bootstrap starting.
I20260927 15:06:35.246609 2653 tablet_bootstrap.cc:492] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: Bootstrap starting.
I20260927 15:06:35.246759 2651 tablet_bootstrap.cc:654] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.246765 2652 tablet_bootstrap.cc:492] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: Bootstrap starting.
I20260927 15:06:35.248507 2652 tablet_bootstrap.cc:654] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.248562 2653 tablet_bootstrap.cc:654] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.249648 2651 tablet_bootstrap.cc:492] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: No bootstrap required, opened a new log
I20260927 15:06:35.249851 2651 ts_tablet_manager.cc:1397] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:06:35.250936 2651 raft_consensus.cc:348] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.251341 2651 raft_consensus.cc:374] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.251459 2651 raft_consensus.cc:729] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.252216 2651 consensus_queue.cc:260] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.252827 2651 ts_tablet_manager.cc:1428] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:35.252633 2657 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.253551 2657 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.254333 2652 tablet_bootstrap.cc:492] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: No bootstrap required, opened a new log
I20260927 15:06:35.254485 2652 ts_tablet_manager.cc:1397] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 15:06:35.255281 2652 raft_consensus.cc:348] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.255501 2652 raft_consensus.cc:374] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.255611 2652 raft_consensus.cc:729] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.255867 2652 consensus_queue.cc:260] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.256238 2653 tablet_bootstrap.cc:492] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: No bootstrap required, opened a new log
I20260927 15:06:35.256400 2653 ts_tablet_manager.cc:1397] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:06:35.256356 2659 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.256743 2659 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.257180 2652 ts_tablet_manager.cc:1428] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:06:35.257184 2653 raft_consensus.cc:348] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.257467 2653 raft_consensus.cc:374] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.257568 2653 raft_consensus.cc:729] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12388ef6606048d4900d86655b8babe3, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.257835 2653 consensus_queue.cc:260] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.258317 2658 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.258607 2658 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.259052 2653 ts_tablet_manager.cc:1428] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 15:06:35.264654 2354 tablet.cc:2369] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:35.333756 2659 raft_consensus.cc:482] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:35.334033 2659 raft_consensus.cc:504] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.335670 2659 leader_election.cc:288] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249), bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831)
I20260927 15:06:35.343118 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.336183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40021 (local address 127.0.97.3:40831)
0927 15:06:35.336409 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.336419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.337127 (+ 708us) server_negotiation.cc:407] Connection header received
0927 15:06:35.337329 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.337336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.337398 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.337494 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:35.338168 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.338896 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.341067 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.342068 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.342147 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.342227 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.342675 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.342803 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:06:35.343447 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.336023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40831 (local address 127.0.97.1:40021)
0927 15:06:35.337023 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.337042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.337180 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.337518 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.337522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.337546 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:35.337972 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.337985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.339034 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.339041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.340878 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.340896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.342376 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.342380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.342511 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.342532 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.342648 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.342821 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":90,"thread_start_us":542,"threads_started":1}
I20260927 15:06:35.343494 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.336126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45747 (local address 127.0.97.2:33249)
0927 15:06:35.336353 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.336363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.337037 (+ 674us) server_negotiation.cc:407] Connection header received
0927 15:06:35.337270 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.337274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.337349 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.337456 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:35.338205 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.338960 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.341050 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.342322 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.342499 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.342603 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.342999 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.343169 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:06:35.343802 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.336015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33249 (local address 127.0.97.1:45747)
0927 15:06:35.336904 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.336920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.337068 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.337471 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.337476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.337501 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:35.338030 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.338040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.339137 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.339145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.340894 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.340907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.342568 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.342574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.342815 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.342840 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.342969 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.343131 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":89,"thread_start_us":639,"threads_started":1}
I20260927 15:06:35.344518 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bbf729c6f48729297c0d2efa9e4d8" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3" is_pre_election: true
I20260927 15:06:35.344591 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bbf729c6f48729297c0d2efa9e4d8" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb9e58b54744290bef288ece3234b7e" is_pre_election: true
I20260927 15:06:35.344826 2443 raft_consensus.cc:2454] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 0.
I20260927 15:06:35.344974 2582 raft_consensus.cc:2454] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 0.
I20260927 15:06:35.345346 2236 leader_election.cc:302] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12388ef6606048d4900d86655b8babe3, b749d353c10741e6bdde4b8a9a58d49b; no voters:
I20260927 15:06:35.345759 2659 raft_consensus.cc:2781] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:35.345897 2659 raft_consensus.cc:482] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:35.346010 2659 raft_consensus.cc:3037] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.348280 2659 raft_consensus.cc:504] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.349004 2659 leader_election.cc:288] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 election: Requested vote from peers 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249), bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831)
I20260927 15:06:35.349628 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bbf729c6f48729297c0d2efa9e4d8" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb9e58b54744290bef288ece3234b7e"
I20260927 15:06:35.349623 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bbf729c6f48729297c0d2efa9e4d8" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3"
I20260927 15:06:35.349818 2582 raft_consensus.cc:3037] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.349866 2443 raft_consensus.cc:3037] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.352396 2582 raft_consensus.cc:2454] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 1.
I20260927 15:06:35.352934 2234 leader_election.cc:302] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:35.352995 2443 raft_consensus.cc:2454] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 1.
I20260927 15:06:35.353276 2659 raft_consensus.cc:2781] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:35.353816 2659 raft_consensus.cc:686] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [term 1 LEADER]: Becoming Leader. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Running, Role: LEADER
I20260927 15:06:35.354372 2659 consensus_queue.cc:237] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.357967 2674 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: TxnStatusTablet state changed. Reason: New leader b749d353c10741e6bdde4b8a9a58d49b. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.358426 2674 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:06:35.359222 2676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:06:35.359534 2676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:06:35.359797 2144 catalog_manager.cc:5495] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b reported cstate change: term changed from 0 to 1, leader changed from <none> to b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } health_report { overall_health: UNKNOWN } } }
I20260927 15:06:35.381763 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.375786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35721 (local address 127.0.0.1:36122)
0927 15:06:35.376060 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.376076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.376248 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.376744 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.376748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.376762 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:06:35.376953 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.376961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.378024 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.378031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.379120 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.379139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.379935 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.379943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.380112 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.380202 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.381169 (+ 967us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.381194 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.381282 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.381448 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 15:06:35.381772 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.375782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.0.97.1:35721)
0927 15:06:35.376011 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.376018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.376155 (+ 137us) server_negotiation.cc:407] Connection header received
0927 15:06:35.376462 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.376465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.376560 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.376713 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:35.377083 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.377892 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.379281 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.379736 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.379872 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.380422 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.381017 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.381203 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.381311 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.381394 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 15:06:35.391543 2582 raft_consensus.cc:1261] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Refusing update from remote peer b749d353c10741e6bdde4b8a9a58d49b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:35.391582 2443 raft_consensus.cc:1261] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Refusing update from remote peer b749d353c10741e6bdde4b8a9a58d49b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:35.392611 2674 consensus_queue.cc:1035] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:35.393015 2659 consensus_queue.cc:1035] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:35.409173 2657 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: TxnStatusTablet state changed. Reason: New leader b749d353c10741e6bdde4b8a9a58d49b. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.409497 2657 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.412509 2659 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.412829 2659 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:06:35.414252 2680 mvcc.cc:205] Tried to move back new op lower bound from 7333976454711930880 to 7333976454576025600. Current Snapshot: MvccSnapshot[applied={T|T < 7333976454711930880}]
I20260927 15:06:35.414494 2658 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: TxnStatusTablet state changed. Reason: New leader b749d353c10741e6bdde4b8a9a58d49b. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.414798 2658 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.417604 2674 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.417954 2674 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P b749d353c10741e6bdde4b8a9a58d49b: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:06:35.426744 2657 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.427160 2657 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P bfb9e58b54744290bef288ece3234b7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.430372 2658 tablet_replica.cc:439] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } } }
I20260927 15:06:35.430701 2658 tablet_replica.cc:442] T 2c7bbf729c6f48729297c0d2efa9e4d8 P 12388ef6606048d4900d86655b8babe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:06:35.438812 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.432154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.0.97.3:40831)
0927 15:06:35.432365 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.432371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.432452 (+ 81us) server_negotiation.cc:407] Connection header received
0927 15:06:35.432756 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.432762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.432855 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.433020 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:35.433432 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.434341 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.436405 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.436838 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.436963 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.437529 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.438097 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.438216 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.438370 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.438445 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 15:06:35.457571 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.450774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48785 (local address 127.0.97.1:35721)
0927 15:06:35.451009 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.451015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.451527 (+ 512us) server_negotiation.cc:407] Connection header received
0927 15:06:35.451742 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.451746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.451833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.451914 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:35.452570 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.453468 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.455555 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.456060 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.456174 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.456863 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.456979 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.457058 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.457232 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.457322 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 15:06:35.457772 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.450584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35721 (local address 127.0.97.3:48785)
0927 15:06:35.451406 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.451428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.451601 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.451956 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.451962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.451976 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:06:35.452301 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.452312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.453613 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.453621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.455389 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.455407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.456262 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.456272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.456460 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.456544 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.457087 (+ 543us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.457113 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.457203 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.457360 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":342,"threads_started":1}
I20260927 15:06:35.469066 2443 raft_consensus.cc:1261] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Refusing update from remote peer bfb9e58b54744290bef288ece3234b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:35.469273 2307 raft_consensus.cc:1261] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Refusing update from remote peer bfb9e58b54744290bef288ece3234b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:35.469859 2661 consensus_queue.cc:1035] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:35.470293 2657 consensus_queue.cc:1035] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:35.515143 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.507393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34647 (local address 127.0.97.3:40831)
0927 15:06:35.507620 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.507630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.508668 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 15:06:35.508907 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.508916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.509005 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.509115 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:35.509768 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.510559 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.512843 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.513559 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.513699 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.514378 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.514514 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.514610 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.514712 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.514803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:06:35.515297 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.507315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40831 (local address 127.0.97.1:34647)
0927 15:06:35.508538 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.508561 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.508735 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.509139 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.509145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.509158 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:06:35.509586 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.509598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.510698 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.510704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.512679 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.512695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.513892 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.513899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.514093 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.514185 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.514618 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.514627 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.514690 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.514847 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":673,"threads_started":1}
I20260927 15:06:35.559851 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:35.560173 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:35.564749 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Sending DeleteTablet for 3 replicas of tablet 4bbe61dc2f7a46228982e9447d3201ec
I20260927 15:06:35.566107 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bbe61dc2f7a46228982e9447d3201ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47686
I20260927 15:06:35.566210 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bbe61dc2f7a46228982e9447d3201ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50702
I20260927 15:06:35.566613 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bbe61dc2f7a46228982e9447d3201ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36118
I20260927 15:06:35.569232 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:35.570030 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:35.578347 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca0574fa97b431c929494d55974673d (DEFAULT_TABLE table=test-table [id=74d3de933a284f9ebf7f1e9cf4d5634a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:35.578487 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca0574fa97b431c929494d55974673d (DEFAULT_TABLE table=test-table [id=74d3de933a284f9ebf7f1e9cf4d5634a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:35.579151 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca0574fa97b431c929494d55974673d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.579164 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca0574fa97b431c929494d55974673d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.584307 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca0574fa97b431c929494d55974673d (DEFAULT_TABLE table=test-table [id=74d3de933a284f9ebf7f1e9cf4d5634a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:35.585111 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca0574fa97b431c929494d55974673d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.585502 2698 tablet_replica.cc:330] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: stopping tablet replica
I20260927 15:06:35.586042 2698 raft_consensus.cc:2229] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:35.586503 2698 raft_consensus.cc:2258] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:35.587498 2653 tablet_bootstrap.cc:492] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Bootstrap starting.
I20260927 15:06:35.589003 2698 ts_tablet_manager.cc:1905] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:35.589304 2653 tablet_bootstrap.cc:654] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.599852 2653 tablet_bootstrap.cc:492] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: No bootstrap required, opened a new log
I20260927 15:06:35.600054 2653 ts_tablet_manager.cc:1397] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260927 15:06:35.600097 2698 ts_tablet_manager.cc:1918] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:35.600418 2698 log.cc:1196] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/4bbe61dc2f7a46228982e9447d3201ec
I20260927 15:06:35.601083 2698 ts_tablet_manager.cc:1939] T 4bbe61dc2f7a46228982e9447d3201ec P 12388ef6606048d4900d86655b8babe3: Deleting consensus metadata
I20260927 15:06:35.602507 2652 tablet_bootstrap.cc:492] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Bootstrap starting.
I20260927 15:06:35.601078 2653 raft_consensus.cc:348] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.602816 2653 raft_consensus.cc:374] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.602957 2653 raft_consensus.cc:729] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12388ef6606048d4900d86655b8babe3, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.603328 2653 consensus_queue.cc:260] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.604040 2653 ts_tablet_manager.cc:1428] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:35.604631 2652 tablet_bootstrap.cc:654] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.605661 2133 catalog_manager.cc:4841] TS 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249): tablet 4bbe61dc2f7a46228982e9447d3201ec (table test-table [id=b527130737ab45c3a7fbb38971e92126]) successfully deleted
I20260927 15:06:35.609704 2651 tablet_bootstrap.cc:492] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Bootstrap starting.
I20260927 15:06:35.611976 2651 tablet_bootstrap.cc:654] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.615202 2651 tablet_bootstrap.cc:492] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: No bootstrap required, opened a new log
I20260927 15:06:35.615475 2651 ts_tablet_manager.cc:1397] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:06:35.615844 2652 tablet_bootstrap.cc:492] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: No bootstrap required, opened a new log
I20260927 15:06:35.616063 2652 ts_tablet_manager.cc:1397] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260927 15:06:35.616525 2651 raft_consensus.cc:348] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.616874 2651 raft_consensus.cc:374] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.617012 2651 raft_consensus.cc:729] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.617365 2651 consensus_queue.cc:260] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.617427 2652 raft_consensus.cc:348] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.617722 2652 raft_consensus.cc:374] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.617852 2652 raft_consensus.cc:729] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.618218 2652 consensus_queue.cc:260] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.619509 2651 ts_tablet_manager.cc:1428] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:06:35.620253 2652 ts_tablet_manager.cc:1428] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:35.621366 2699 tablet_replica.cc:330] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: stopping tablet replica
I20260927 15:06:35.621879 2699 raft_consensus.cc:2229] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:35.624951 2699 raft_consensus.cc:2258] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:35.626971 2691 raft_consensus.cc:482] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:35.627257 2691 raft_consensus.cc:504] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.628407 2691 leader_election.cc:288] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249), bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831)
I20260927 15:06:35.628758 2700 tablet_replica.cc:330] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: stopping tablet replica
I20260927 15:06:35.629290 2700 raft_consensus.cc:2229] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:35.629278 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0574fa97b431c929494d55974673d" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3" is_pre_election: true
I20260927 15:06:35.629591 2442 raft_consensus.cc:2454] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 0.
I20260927 15:06:35.629669 2700 raft_consensus.cc:2258] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:35.630123 2236 leader_election.cc:302] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12388ef6606048d4900d86655b8babe3, b749d353c10741e6bdde4b8a9a58d49b; no voters:
I20260927 15:06:35.630712 2699 ts_tablet_manager.cc:1905] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:35.631029 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0574fa97b431c929494d55974673d" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb9e58b54744290bef288ece3234b7e" is_pre_election: true
I20260927 15:06:35.631438 2581 raft_consensus.cc:2454] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 0.
I20260927 15:06:35.632246 2691 raft_consensus.cc:2781] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:35.632402 2691 raft_consensus.cc:482] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:35.632531 2691 raft_consensus.cc:3037] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.633955 2700 ts_tablet_manager.cc:1905] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:35.638830 2691 raft_consensus.cc:504] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.640172 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0574fa97b431c929494d55974673d" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3"
I20260927 15:06:35.640453 2442 raft_consensus.cc:3037] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.643972 2442 raft_consensus.cc:2454] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 1.
I20260927 15:06:35.645412 2236 leader_election.cc:302] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12388ef6606048d4900d86655b8babe3, b749d353c10741e6bdde4b8a9a58d49b; no voters:
I20260927 15:06:35.645859 2659 raft_consensus.cc:2781] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:35.646024 2659 raft_consensus.cc:686] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 1 LEADER]: Becoming Leader. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Running, Role: LEADER
I20260927 15:06:35.646341 2659 consensus_queue.cc:237] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:35.648945 2699 ts_tablet_manager.cc:1918] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:35.649184 2699 log.cc:1196] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/4bbe61dc2f7a46228982e9447d3201ec
I20260927 15:06:35.650013 2699 ts_tablet_manager.cc:1939] T 4bbe61dc2f7a46228982e9447d3201ec P bfb9e58b54744290bef288ece3234b7e: Deleting consensus metadata
I20260927 15:06:35.651696 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca0574fa97b431c929494d55974673d" candidate_uuid: "b749d353c10741e6bdde4b8a9a58d49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb9e58b54744290bef288ece3234b7e"
I20260927 15:06:35.652009 2581 raft_consensus.cc:3037] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.653940 2700 ts_tablet_manager.cc:1918] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:35.654145 2700 log.cc:1196] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/4bbe61dc2f7a46228982e9447d3201ec
I20260927 15:06:35.654868 2700 ts_tablet_manager.cc:1939] T 4bbe61dc2f7a46228982e9447d3201ec P b749d353c10741e6bdde4b8a9a58d49b: Deleting consensus metadata
I20260927 15:06:35.657004 2132 catalog_manager.cc:4841] TS b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721): tablet 4bbe61dc2f7a46228982e9447d3201ec (table test-table [id=b527130737ab45c3a7fbb38971e92126]) successfully deleted
I20260927 15:06:35.658537 2581 raft_consensus.cc:2454] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b749d353c10741e6bdde4b8a9a58d49b in term 1.
I20260927 15:06:35.659467 2131 catalog_manager.cc:4841] TS bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831): tablet 4bbe61dc2f7a46228982e9447d3201ec (table test-table [id=b527130737ab45c3a7fbb38971e92126]) successfully deleted
I20260927 15:06:35.660218 2691 leader_election.cc:288] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [CANDIDATE]: Term 1 election: Requested vote from peers 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249), bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831)
I20260927 15:06:35.661252 2144 catalog_manager.cc:5495] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b reported cstate change: term changed from 0 to 1, leader changed from <none> to b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b749d353c10741e6bdde4b8a9a58d49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } health_report { overall_health: UNKNOWN } } }
I20260927 15:06:35.687493 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.682108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.0.97.1:35721)
0927 15:06:35.682297 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.682308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.682324 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:06:35.682377 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.682381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.682442 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.682558 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:35.683050 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.683994 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.685411 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.685944 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.686113 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.686555 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.686720 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.686841 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.687003 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.687122 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:06:35.702229 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.696828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43473 (local address 127.0.97.1:35721)
0927 15:06:35.697038 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.697046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.697282 (+ 236us) server_negotiation.cc:407] Connection header received
0927 15:06:35.697509 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.697514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.697580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.697682 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:35.698060 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.698882 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.700394 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.700859 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.700994 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.701401 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.701539 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.701651 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.701775 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.701848 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":60}
I20260927 15:06:35.702229 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.696875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35721 (local address 127.0.97.1:43473)
0927 15:06:35.697188 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:35.697204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:35.697357 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:35.697691 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:35.697696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:35.697707 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:06:35.697931 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.697939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.699029 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.699036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.700236 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:35.700249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:35.701025 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:35.701032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:35.701190 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:35.701226 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:06:35.701651 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:06:35.701663 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:06:35.701748 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:06:35.701901 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":23}
I20260927 15:06:35.715294 2442 raft_consensus.cc:1261] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Refusing update from remote peer b749d353c10741e6bdde4b8a9a58d49b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:35.715312 2581 raft_consensus.cc:1261] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Refusing update from remote peer b749d353c10741e6bdde4b8a9a58d49b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:35.715982 2659 consensus_queue.cc:1035] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:35.716588 2691 consensus_queue.cc:1035] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:35.732832 2682 mvcc.cc:205] Tried to move back new op lower bound from 7333976456044462080 to 7333976455770546176. Current Snapshot: MvccSnapshot[applied={T|T < 7333976456044462080}]
I20260927 15:06:35.738539 2679 mvcc.cc:205] Tried to move back new op lower bound from 7333976456044462080 to 7333976455770546176. Current Snapshot: MvccSnapshot[applied={T|T < 7333976456044462080}]
W20260927 15:06:35.769345 2679 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:06:37.413457 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:37.413803 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:37.418248 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Sending DeleteTablet for 3 replicas of tablet 7ca0574fa97b431c929494d55974673d
I20260927 15:06:37.419202 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca0574fa97b431c929494d55974673d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36118
I20260927 15:06:37.419257 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca0574fa97b431c929494d55974673d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47686
I20260927 15:06:37.419309 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca0574fa97b431c929494d55974673d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50702
I20260927 15:06:37.432276 2729 tablet_replica.cc:330] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: stopping tablet replica
I20260927 15:06:37.432580 2729 raft_consensus.cc:2229] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:37.432916 2729 raft_consensus.cc:2258] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:37.433584 2728 tablet_replica.cc:330] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: stopping tablet replica
I20260927 15:06:37.433897 2730 tablet_replica.cc:330] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: stopping tablet replica
I20260927 15:06:37.433876 2728 raft_consensus.cc:2229] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:37.434083 2730 raft_consensus.cc:2229] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:37.434332 2730 raft_consensus.cc:2258] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:37.434468 2728 raft_consensus.cc:2258] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:37.434754 2729 ts_tablet_manager.cc:1905] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:37.435472 2728 ts_tablet_manager.cc:1905] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:37.435472 2730 ts_tablet_manager.cc:1905] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:37.442425 2730 ts_tablet_manager.cc:1918] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:06:37.442425 2728 ts_tablet_manager.cc:1918] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:06:37.442591 2728 log.cc:1196] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/7ca0574fa97b431c929494d55974673d
I20260927 15:06:37.442705 2730 log.cc:1196] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/7ca0574fa97b431c929494d55974673d
I20260927 15:06:37.442723 2729 ts_tablet_manager.cc:1918] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:06:37.442963 2729 log.cc:1196] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/7ca0574fa97b431c929494d55974673d
I20260927 15:06:37.443187 2728 ts_tablet_manager.cc:1939] T 7ca0574fa97b431c929494d55974673d P b749d353c10741e6bdde4b8a9a58d49b: Deleting consensus metadata
I20260927 15:06:37.443212 2730 ts_tablet_manager.cc:1939] T 7ca0574fa97b431c929494d55974673d P bfb9e58b54744290bef288ece3234b7e: Deleting consensus metadata
I20260927 15:06:37.443516 2729 ts_tablet_manager.cc:1939] T 7ca0574fa97b431c929494d55974673d P 12388ef6606048d4900d86655b8babe3: Deleting consensus metadata
I20260927 15:06:37.444669 2132 catalog_manager.cc:4841] TS b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721): tablet 7ca0574fa97b431c929494d55974673d (table test-table [id=74d3de933a284f9ebf7f1e9cf4d5634a]) successfully deleted
I20260927 15:06:37.444860 2133 catalog_manager.cc:4841] TS 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249): tablet 7ca0574fa97b431c929494d55974673d (table test-table [id=74d3de933a284f9ebf7f1e9cf4d5634a]) successfully deleted
I20260927 15:06:37.445585 2131 catalog_manager.cc:4841] TS bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831): tablet 7ca0574fa97b431c929494d55974673d (table test-table [id=74d3de933a284f9ebf7f1e9cf4d5634a]) successfully deleted
Sep 27 15:06:37 dist-test-slave-g0w1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:06:37 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:37.480549 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:37.464754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.0.97.62:34015)
0927 15:06:37.465284 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:37.465289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:37.465320 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:06:37.465477 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:37.465481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:37.465552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:37.465661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:37.467001 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.467880 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.469784 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.470213 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.470342 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:37.474699 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:37.474707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:37.474711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:37.474758 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:37.477706 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:37.478491 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:37.478496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:37.478498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:37.478667 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:37.479048 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:37.479051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:37.479053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:37.479450 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:37.479632 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:37.479968 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 15:06:37.480274 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
I20260927 15:06:37.484407 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:37.485100 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:37.491983 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 37a59eea70b4489cb537d20ba0930005 (DEFAULT_TABLE table=test-table [id=42bb1282c342402698d2a26a3e09473d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:37.492065 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 37a59eea70b4489cb537d20ba0930005 (DEFAULT_TABLE table=test-table [id=42bb1282c342402698d2a26a3e09473d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:37.492213 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 37a59eea70b4489cb537d20ba0930005 (DEFAULT_TABLE table=test-table [id=42bb1282c342402698d2a26a3e09473d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:37.492497 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a59eea70b4489cb537d20ba0930005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.492522 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a59eea70b4489cb537d20ba0930005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.492612 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a59eea70b4489cb537d20ba0930005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.499941 2739 tablet_bootstrap.cc:492] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Bootstrap starting.
I20260927 15:06:37.501322 2739 tablet_bootstrap.cc:654] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.503641 2741 tablet_bootstrap.cc:492] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Bootstrap starting.
I20260927 15:06:37.504120 2740 tablet_bootstrap.cc:492] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Bootstrap starting.
I20260927 15:06:37.505753 2741 tablet_bootstrap.cc:654] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.506367 2740 tablet_bootstrap.cc:654] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.507493 2739 tablet_bootstrap.cc:492] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: No bootstrap required, opened a new log
I20260927 15:06:37.507758 2739 ts_tablet_manager.cc:1397] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260927 15:06:37.508867 2739 raft_consensus.cc:348] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.509097 2739 raft_consensus.cc:374] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.509203 2739 raft_consensus.cc:729] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12388ef6606048d4900d86655b8babe3, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.509212 2740 tablet_bootstrap.cc:492] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: No bootstrap required, opened a new log
I20260927 15:06:37.509394 2740 ts_tablet_manager.cc:1397] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 15:06:37.509491 2739 consensus_queue.cc:260] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.509881 2741 tablet_bootstrap.cc:492] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: No bootstrap required, opened a new log
I20260927 15:06:37.510099 2741 ts_tablet_manager.cc:1397] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260927 15:06:37.510353 2739 ts_tablet_manager.cc:1428] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:06:37.510260 2740 raft_consensus.cc:348] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.510519 2740 raft_consensus.cc:374] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.510630 2740 raft_consensus.cc:729] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.510892 2740 consensus_queue.cc:260] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.511572 2740 ts_tablet_manager.cc:1428] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:37.511670 2741 raft_consensus.cc:348] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.511919 2741 raft_consensus.cc:374] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.512030 2741 raft_consensus.cc:729] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.512282 2741 consensus_queue.cc:260] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.515692 2741 ts_tablet_manager.cc:1428] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:06:37.521778 2746 raft_consensus.cc:482] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:37.522078 2746 raft_consensus.cc:504] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.522926 2746 leader_election.cc:288] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721), 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249)
I20260927 15:06:37.523592 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a59eea70b4489cb537d20ba0930005" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3" is_pre_election: true
I20260927 15:06:37.523592 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a59eea70b4489cb537d20ba0930005" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b749d353c10741e6bdde4b8a9a58d49b" is_pre_election: true
I20260927 15:06:37.523871 2443 raft_consensus.cc:2454] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 0.
I20260927 15:06:37.523871 2306 raft_consensus.cc:2454] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 0.
I20260927 15:06:37.524420 2515 leader_election.cc:302] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:37.524753 2746 raft_consensus.cc:2781] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:37.524880 2746 raft_consensus.cc:482] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:37.524991 2746 raft_consensus.cc:3037] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.528159 2746 raft_consensus.cc:504] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.528788 2746 leader_election.cc:288] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 election: Requested vote from peers b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721), 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249)
I20260927 15:06:37.529307 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a59eea70b4489cb537d20ba0930005" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3"
I20260927 15:06:37.529338 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a59eea70b4489cb537d20ba0930005" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b749d353c10741e6bdde4b8a9a58d49b"
I20260927 15:06:37.529526 2443 raft_consensus.cc:3037] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.529543 2306 raft_consensus.cc:3037] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.531818 2306 raft_consensus.cc:2454] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 1.
I20260927 15:06:37.532021 2443 raft_consensus.cc:2454] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 1.
I20260927 15:06:37.532400 2515 leader_election.cc:302] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:37.532727 2746 raft_consensus.cc:2781] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:37.533177 2746 raft_consensus.cc:686] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 1 LEADER]: Becoming Leader. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Running, Role: LEADER
I20260927 15:06:37.533471 2746 consensus_queue.cc:237] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.536455 2146 catalog_manager.cc:5495] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb9e58b54744290bef288ece3234b7e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bfb9e58b54744290bef288ece3234b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } health_report { overall_health: HEALTHY } } }
I20260927 15:06:37.566390 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:37.560655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.97.3:40831)
0927 15:06:37.561260 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:37.561264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:37.561283 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:06:37.561338 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:37.561342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:37.561403 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:37.561515 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:37.562384 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.563251 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.564572 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.565012 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.565155 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:37.565599 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:37.565710 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:37.565810 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:37.565956 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:06:37.566036 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":321,"threads_started":1}
I20260927 15:06:37.575736 2443 raft_consensus.cc:1261] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Refusing update from remote peer bfb9e58b54744290bef288ece3234b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:37.575912 2306 raft_consensus.cc:1261] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Refusing update from remote peer bfb9e58b54744290bef288ece3234b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:37.576363 2747 consensus_queue.cc:1035] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:37.576756 2746 consensus_queue.cc:1035] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:37.625756 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:37.626050 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:37.629499 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Sending DeleteTablet for 3 replicas of tablet 37a59eea70b4489cb537d20ba0930005
I20260927 15:06:37.630410 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a59eea70b4489cb537d20ba0930005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36118
I20260927 15:06:37.630534 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a59eea70b4489cb537d20ba0930005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47686
I20260927 15:06:37.630712 2728 tablet_replica.cc:330] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: stopping tablet replica
I20260927 15:06:37.630708 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a59eea70b4489cb537d20ba0930005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50702
I20260927 15:06:37.630817 2729 tablet_replica.cc:330] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: stopping tablet replica
I20260927 15:06:37.630924 2728 raft_consensus.cc:2229] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:37.631003 2730 tablet_replica.cc:330] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: stopping tablet replica
I20260927 15:06:37.631011 2729 raft_consensus.cc:2229] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:37.631175 2728 raft_consensus.cc:2258] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:37.631697 2729 raft_consensus.cc:2258] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:37.631680 2730 raft_consensus.cc:2229] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:37.631995 2730 raft_consensus.cc:2258] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:37.632423 2728 ts_tablet_manager.cc:1905] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:37.632429 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:06:37.633087 2730 ts_tablet_manager.cc:1905] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:06:37.633157 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:37.633268 2729 ts_tablet_manager.cc:1905] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:37.638374 2728 ts_tablet_manager.cc:1918] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:37.638558 2728 log.cc:1196] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/37a59eea70b4489cb537d20ba0930005
I20260927 15:06:37.639202 2728 ts_tablet_manager.cc:1939] T 37a59eea70b4489cb537d20ba0930005 P b749d353c10741e6bdde4b8a9a58d49b: Deleting consensus metadata
I20260927 15:06:37.640410 2132 catalog_manager.cc:4841] TS b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721): tablet 37a59eea70b4489cb537d20ba0930005 (table test-table [id=42bb1282c342402698d2a26a3e09473d]) successfully deleted
I20260927 15:06:37.640640 2730 ts_tablet_manager.cc:1918] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:37.640878 2730 log.cc:1196] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/37a59eea70b4489cb537d20ba0930005
I20260927 15:06:37.641353 2730 ts_tablet_manager.cc:1939] T 37a59eea70b4489cb537d20ba0930005 P bfb9e58b54744290bef288ece3234b7e: Deleting consensus metadata
I20260927 15:06:37.641855 2729 ts_tablet_manager.cc:1918] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:06:37.642057 2729 log.cc:1196] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/37a59eea70b4489cb537d20ba0930005
I20260927 15:06:37.642472 2729 ts_tablet_manager.cc:1939] T 37a59eea70b4489cb537d20ba0930005 P 12388ef6606048d4900d86655b8babe3: Deleting consensus metadata
I20260927 15:06:37.642450 2131 catalog_manager.cc:4841] TS bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831): tablet 37a59eea70b4489cb537d20ba0930005 (table test-table [id=42bb1282c342402698d2a26a3e09473d]) successfully deleted
I20260927 15:06:37.643586 2133 catalog_manager.cc:4841] TS 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249): tablet 37a59eea70b4489cb537d20ba0930005 (table test-table [id=42bb1282c342402698d2a26a3e09473d]) successfully deleted
I20260927 15:06:37.644354 2282 tablet_service.cc:1467] Processing CreateTablet for tablet b91f98a9363045d3a3c6bfa5e29ef048 (DEFAULT_TABLE table=test-table [id=ed36773b18174665999b6ccf99322aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:37.644666 2422 tablet_service.cc:1467] Processing CreateTablet for tablet b91f98a9363045d3a3c6bfa5e29ef048 (DEFAULT_TABLE table=test-table [id=ed36773b18174665999b6ccf99322aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:37.644788 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91f98a9363045d3a3c6bfa5e29ef048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.644754 2561 tablet_service.cc:1467] Processing CreateTablet for tablet b91f98a9363045d3a3c6bfa5e29ef048 (DEFAULT_TABLE table=test-table [id=ed36773b18174665999b6ccf99322aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:37.645056 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91f98a9363045d3a3c6bfa5e29ef048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.645184 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91f98a9363045d3a3c6bfa5e29ef048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.649861 2741 tablet_bootstrap.cc:492] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Bootstrap starting.
I20260927 15:06:37.651625 2741 tablet_bootstrap.cc:654] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.651777 2740 tablet_bootstrap.cc:492] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Bootstrap starting.
I20260927 15:06:37.651777 2739 tablet_bootstrap.cc:492] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Bootstrap starting.
I20260927 15:06:37.653475 2740 tablet_bootstrap.cc:654] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.653491 2739 tablet_bootstrap.cc:654] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.655885 2739 tablet_bootstrap.cc:492] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: No bootstrap required, opened a new log
I20260927 15:06:37.656078 2739 ts_tablet_manager.cc:1397] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 15:06:37.657044 2740 tablet_bootstrap.cc:492] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: No bootstrap required, opened a new log
I20260927 15:06:37.656883 2739 raft_consensus.cc:348] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.657145 2739 raft_consensus.cc:374] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.657265 2739 raft_consensus.cc:729] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12388ef6606048d4900d86655b8babe3, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.657322 2740 ts_tablet_manager.cc:1397] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 15:06:37.657581 2739 consensus_queue.cc:260] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.658211 2739 ts_tablet_manager.cc:1428] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:06:37.658954 2740 raft_consensus.cc:348] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.659443 2740 raft_consensus.cc:374] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.659559 2740 raft_consensus.cc:729] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b749d353c10741e6bdde4b8a9a58d49b, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.659847 2740 consensus_queue.cc:260] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.660308 2741 tablet_bootstrap.cc:492] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: No bootstrap required, opened a new log
I20260927 15:06:37.660517 2741 ts_tablet_manager.cc:1397] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 15:06:37.661036 2740 ts_tablet_manager.cc:1428] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:06:37.661314 2741 raft_consensus.cc:348] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.661583 2741 raft_consensus.cc:374] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.661680 2741 raft_consensus.cc:729] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.662045 2741 consensus_queue.cc:260] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.662844 2741 ts_tablet_manager.cc:1428] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:06:37.737727 2754 raft_consensus.cc:482] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:37.737934 2754 raft_consensus.cc:504] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.738736 2754 leader_election.cc:288] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721), 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249)
I20260927 15:06:37.739436 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91f98a9363045d3a3c6bfa5e29ef048" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b749d353c10741e6bdde4b8a9a58d49b" is_pre_election: true
I20260927 15:06:37.739432 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91f98a9363045d3a3c6bfa5e29ef048" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3" is_pre_election: true
I20260927 15:06:37.739696 2306 raft_consensus.cc:2454] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 0.
I20260927 15:06:37.739706 2442 raft_consensus.cc:2454] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 0.
I20260927 15:06:37.740182 2515 leader_election.cc:302] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:37.740489 2754 raft_consensus.cc:2781] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:37.740593 2754 raft_consensus.cc:482] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:37.740710 2754 raft_consensus.cc:3037] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.742787 2754 raft_consensus.cc:504] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.743479 2754 leader_election.cc:288] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 election: Requested vote from peers b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721), 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249)
I20260927 15:06:37.743966 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91f98a9363045d3a3c6bfa5e29ef048" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b749d353c10741e6bdde4b8a9a58d49b"
I20260927 15:06:37.744062 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91f98a9363045d3a3c6bfa5e29ef048" candidate_uuid: "bfb9e58b54744290bef288ece3234b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12388ef6606048d4900d86655b8babe3"
I20260927 15:06:37.744166 2306 raft_consensus.cc:3037] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.744323 2442 raft_consensus.cc:3037] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.746248 2306 raft_consensus.cc:2454] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 1.
I20260927 15:06:37.746699 2515 leader_election.cc:302] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b749d353c10741e6bdde4b8a9a58d49b, bfb9e58b54744290bef288ece3234b7e; no voters:
I20260927 15:06:37.746999 2754 raft_consensus.cc:2781] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:37.747195 2754 raft_consensus.cc:686] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 1 LEADER]: Becoming Leader. State: Replica: bfb9e58b54744290bef288ece3234b7e, State: Running, Role: LEADER
I20260927 15:06:37.747437 2442 raft_consensus.cc:2454] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb9e58b54744290bef288ece3234b7e in term 1.
I20260927 15:06:37.747516 2754 consensus_queue.cc:237] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } }
I20260927 15:06:37.750118 2146 catalog_manager.cc:5495] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb9e58b54744290bef288ece3234b7e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bfb9e58b54744290bef288ece3234b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb9e58b54744290bef288ece3234b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40831 } health_report { overall_health: HEALTHY } } }
I20260927 15:06:37.798177 2442 raft_consensus.cc:1261] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Refusing update from remote peer bfb9e58b54744290bef288ece3234b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:37.798656 2306 raft_consensus.cc:1261] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Refusing update from remote peer bfb9e58b54744290bef288ece3234b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:37.798707 2747 consensus_queue.cc:1035] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12388ef6606048d4900d86655b8babe3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:37.799189 2754 consensus_queue.cc:1035] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b749d353c10741e6bdde4b8a9a58d49b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:06:37.819589 2713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:06:39.418269 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:39.418574 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:39.423290 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77db2be6617c4b529674c3f5f70c2f32: Sending DeleteTablet for 3 replicas of tablet b91f98a9363045d3a3c6bfa5e29ef048
I20260927 15:06:39.424223 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet b91f98a9363045d3a3c6bfa5e29ef048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36118
I20260927 15:06:39.424414 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet b91f98a9363045d3a3c6bfa5e29ef048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47686
I20260927 15:06:39.424391 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet b91f98a9363045d3a3c6bfa5e29ef048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50702
I20260927 15:06:39.425856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2217
I20260927 15:06:39.437778 2786 tablet_replica.cc:330] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: stopping tablet replica
I20260927 15:06:39.438105 2786 raft_consensus.cc:2229] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:39.438627 2786 raft_consensus.cc:2258] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:39.439890 2784 tablet_replica.cc:330] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: stopping tablet replica
I20260927 15:06:39.440014 2786 ts_tablet_manager.cc:1905] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:39.440125 2784 raft_consensus.cc:2229] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:39.440389 2784 raft_consensus.cc:2258] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:39.442317 2784 ts_tablet_manager.cc:1905] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:39.442345 2785 tablet_replica.cc:330] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: stopping tablet replica
I20260927 15:06:39.442559 2785 raft_consensus.cc:2229] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:39.442893 2785 raft_consensus.cc:2258] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:39.444356 2785 ts_tablet_manager.cc:1905] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:39.447165 2786 ts_tablet_manager.cc:1918] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:06:39.447399 2786 log.cc:1196] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/b91f98a9363045d3a3c6bfa5e29ef048
I20260927 15:06:39.447924 2786 ts_tablet_manager.cc:1939] T b91f98a9363045d3a3c6bfa5e29ef048 P bfb9e58b54744290bef288ece3234b7e: Deleting consensus metadata
I20260927 15:06:39.450069 2131 catalog_manager.cc:4841] TS bfb9e58b54744290bef288ece3234b7e (127.0.97.3:40831): tablet b91f98a9363045d3a3c6bfa5e29ef048 (table test-table [id=ed36773b18174665999b6ccf99322aad]) successfully deleted
I20260927 15:06:39.450448 2785 ts_tablet_manager.cc:1918] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:06:39.450450 2784 ts_tablet_manager.cc:1918] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:06:39.450604 2785 log.cc:1196] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/b91f98a9363045d3a3c6bfa5e29ef048
I20260927 15:06:39.450610 2784 log.cc:1196] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/b91f98a9363045d3a3c6bfa5e29ef048
I20260927 15:06:39.451011 2784 ts_tablet_manager.cc:1939] T b91f98a9363045d3a3c6bfa5e29ef048 P b749d353c10741e6bdde4b8a9a58d49b: Deleting consensus metadata
I20260927 15:06:39.451049 2785 ts_tablet_manager.cc:1939] T b91f98a9363045d3a3c6bfa5e29ef048 P 12388ef6606048d4900d86655b8babe3: Deleting consensus metadata
I20260927 15:06:39.452347 2133 catalog_manager.cc:4841] TS 12388ef6606048d4900d86655b8babe3 (127.0.97.2:33249): tablet b91f98a9363045d3a3c6bfa5e29ef048 (table test-table [id=ed36773b18174665999b6ccf99322aad]) successfully deleted
I20260927 15:06:39.452347 2132 catalog_manager.cc:4841] TS b749d353c10741e6bdde4b8a9a58d49b (127.0.97.1:35721): tablet b91f98a9363045d3a3c6bfa5e29ef048 (table test-table [id=ed36773b18174665999b6ccf99322aad]) successfully deleted
Sep 27 15:06:39 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:39.463940 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:39.427993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.0.97.1:35721)
0927 15:06:39.441406 (+ 13413us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:39.441411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:39.441435 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:06:39.441493 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:39.441497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:39.441565 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:39.441723 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:39.443722 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:39.444956 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:39.446887 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:39.447592 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:39.447749 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:39.457421 (+ 9672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:39.457438 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:39.457454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:39.457497 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:39.460908 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:39.461740 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:39.461748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:39.461750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:39.461921 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:39.462306 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:39.462309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:39.462311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:39.462675 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:39.462913 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:39.463357 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:06:39.463540 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13274,"thread_start_us":13855,"threads_started":1}
W20260927 15:06:39.464628 2348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40794
W20260927 15:06:39.465248 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:39.465448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 2217
I20260927 15:06:39.491964 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2357
Sep 27 15:06:39 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:39.508780 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:39.492825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.0.97.2:33249)
0927 15:06:39.493312 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:39.493317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:39.493335 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:06:39.493388 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:39.493392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:39.493452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:39.493547 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:39.494790 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:39.495622 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:39.496958 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:39.497407 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:39.497521 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:39.502195 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:39.502213 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:39.502231 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:39.502278 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:39.505835 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:39.506617 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:39.506626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:39.506629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:39.506778 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:39.507283 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:39.507287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:39.507289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:39.507661 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:39.507908 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:39.508218 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:06:39.508399 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":239,"threads_started":1}
W20260927 15:06:39.509595 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58768
W20260927 15:06:39.510299 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:39.510494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 2357
I20260927 15:06:39.528265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2497
Sep 27 15:06:39 dist-test-slave-g0w1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:39.544648 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:39.529121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.97.3:40831)
0927 15:06:39.529690 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:39.529695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:39.529712 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:06:39.529764 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:39.529768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:39.529825 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:39.529955 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:39.531377 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:39.532221 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:39.533441 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:39.533847 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:39.533981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:39.538680 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:39.538696 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:39.538713 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:39.538764 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:39.541931 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:39.542635 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:39.542643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:39.542645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:39.542800 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:39.543242 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:39.543246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:39.543249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:39.543604 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:39.543896 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:39.544193 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 15:06:39.544377 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":287,"threads_started":1}
W20260927 15:06:39.545323 2627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008
W20260927 15:06:39.546058 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:39.546232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 2497
I20260927 15:06:39.568130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2114
I20260927 15:06:39.569520 2186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:39.688652 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 2114
2026-09-27T15:06:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8391 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:39 dist-test-slave-g0w1 krb5kdc[2794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:39 dist-test-slave-g0w1 krb5kdc[2794](info): set up 2 sockets
Sep 27 15:06:39 dist-test-slave-g0w1 krb5kdc[2794](info): commencing operation
krb5kdc: starting...
W20260927 15:06:40.795588 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:40 dist-test-slave-g0w1 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:41.003913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38075
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:41.339452 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:41.339766 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:41.339927 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:41.347452 2807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:41.347572 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:41.347652 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:41.347757 2807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:41.347851 2807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:41.358145 2807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:41.359601 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:41.360997 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:41.374119 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:41.374187 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:41.374151 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:41.374895 2807 server_base.cc:1034] running on GCE node
I20260927 15:06:41.375837 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:41.377373 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:41.378607 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790521601378561 us; error 41 us; skew 500 ppm
I20260927 15:06:41.378942 2807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:41 dist-test-slave-g0w1 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:41.382266 2807 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:41.383950 2807 webserver.cc:460] Webserver started at http://127.0.97.62:46789/ using document root <none> and password file <none>
I20260927 15:06:41.384661 2807 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:41.384768 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:41.385042 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:41.388190 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "5b79ff85392d47a2a45cf8eed7805c44"
format_stamp: "Formatted at 2026-09-27 15:06:41 on dist-test-slave-g0w1"
server_key: "31fc2006382d20298fd051ee61fe4dc4"
server_key_iv: "c925b1c5921abb0847d6b31879202604"
server_key_version: "encryptionkey@0"
I20260927 15:06:41.389001 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "5b79ff85392d47a2a45cf8eed7805c44"
format_stamp: "Formatted at 2026-09-27 15:06:41 on dist-test-slave-g0w1"
server_key: "31fc2006382d20298fd051ee61fe4dc4"
server_key_iv: "c925b1c5921abb0847d6b31879202604"
server_key_version: "encryptionkey@0"
I20260927 15:06:41.394647 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:41.398578 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:41.400208 2807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:41.400427 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "5b79ff85392d47a2a45cf8eed7805c44"
format_stamp: "Formatted at 2026-09-27 15:06:41 on dist-test-slave-g0w1"
server_key: "31fc2006382d20298fd051ee61fe4dc4"
server_key_iv: "c925b1c5921abb0847d6b31879202604"
server_key_version: "encryptionkey@0"
I20260927 15:06:41.400663 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:41.416428 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:41.420791 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:41.421115 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:41.440534 2807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38075
I20260927 15:06:41.440572 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38075 every 8 connection(s)
I20260927 15:06:41.442052 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:41.444059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 2807
I20260927 15:06:41.444298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:41.445102 388 external_mini_cluster.cc:1428] Setting key 1bd60a2c12070a03a5fa7bc44bd467ee
I20260927 15:06:41.447260 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:06:41 dist-test-slave-g0w1 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:41.461602 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44: Bootstrap starting.
I20260927 15:06:41.466173 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:41.467490 2874 log.cc:824] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:41.471225 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:41.447307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.0.97.62:38075)
0927 15:06:41.448171 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:41.448183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:41.448226 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:06:41.449884 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:41.449907 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:41.450332 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:41.450707 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:41.452614 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:41.454008 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:41.455874 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:41.456618 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:41.456758 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:41.461643 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:41.461656 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:41.461678 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:41.461734 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:41.465925 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:41.466866 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:41.466876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:41.466886 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:41.467158 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:41.467614 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:41.467622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:41.467625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:41.468406 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:41.468642 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:41.469326 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 15:06:41.469821 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":372,"threads_started":1}
I20260927 15:06:41.471309 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44: No bootstrap required, opened a new log
I20260927 15:06:41.479296 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b79ff85392d47a2a45cf8eed7805c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38075 } }
I20260927 15:06:41.479672 2874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:41.479786 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b79ff85392d47a2a45cf8eed7805c44, State: Initialized, Role: FOLLOWER
I20260927 15:06:41.480479 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b79ff85392d47a2a45cf8eed7805c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38075 } }
I20260927 15:06:41.480796 2874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:41.481042 2874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:41.481238 2874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:41.483532 2874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b79ff85392d47a2a45cf8eed7805c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38075 } }
I20260927 15:06:41.484066 2874 leader_election.cc:302] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b79ff85392d47a2a45cf8eed7805c44; no voters:
I20260927 15:06:41.484632 2874 leader_election.cc:288] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:41.484776 2879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:41.485608 2879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [term 1 LEADER]: Becoming Leader. State: Replica: 5b79ff85392d47a2a45cf8eed7805c44, State: Running, Role: LEADER
I20260927 15:06:41.486152 2879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b79ff85392d47a2a45cf8eed7805c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38075 } }
I20260927 15:06:41.487236 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:41.489027 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b79ff85392d47a2a45cf8eed7805c44. Latest consensus state: current_term: 1 leader_uuid: "5b79ff85392d47a2a45cf8eed7805c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b79ff85392d47a2a45cf8eed7805c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38075 } } }
I20260927 15:06:41.489363 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b79ff85392d47a2a45cf8eed7805c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b79ff85392d47a2a45cf8eed7805c44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38075 } } }
I20260927 15:06:41.489552 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:41.489619 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:41.490461 2885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:41.496215 2885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:41.508102 2885 catalog_manager.cc:1347] Generated new cluster ID: be257e253e1f43f78becc5c0afdfdeca
I20260927 15:06:41.508247 2885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:41.523644 2885 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:41.525293 2885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:41.534932 2885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b79ff85392d47a2a45cf8eed7805c44: Generated new TSK 0
I20260927 15:06:41.535924 2885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:41.595228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38075
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:41.975502 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:41.975807 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:41.975909 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:41.983456 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:41.983587 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:41.983701 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:41.994139 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:41.995533 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:41.996838 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:42.011572 2905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:42.011651 2906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:42.011672 2908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:42.012696 2900 server_base.cc:1034] running on GCE node
I20260927 15:06:42.013386 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:42.014469 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:42.015710 2900 hybrid_clock.cc:648] HybridClock initialized: now 1790521602015655 us; error 51 us; skew 500 ppm
I20260927 15:06:42.016098 2900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:42 dist-test-slave-g0w1 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:42.019608 2900 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:42.021335 2900 webserver.cc:460] Webserver started at http://127.0.97.1:42465/ using document root <none> and password file <none>
I20260927 15:06:42.022110 2900 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:42.022275 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:42.022626 2900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:42.025983 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "4accfdb4ab2b4923aecd1030dbc3a608"
format_stamp: "Formatted at 2026-09-27 15:06:42 on dist-test-slave-g0w1"
server_key: "2bb67046ea4d7f3f06c01148950a6d8b"
server_key_iv: "f801e5d86e3f64bac63100c180466e73"
server_key_version: "encryptionkey@0"
I20260927 15:06:42.026928 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4accfdb4ab2b4923aecd1030dbc3a608"
format_stamp: "Formatted at 2026-09-27 15:06:42 on dist-test-slave-g0w1"
server_key: "2bb67046ea4d7f3f06c01148950a6d8b"
server_key_iv: "f801e5d86e3f64bac63100c180466e73"
server_key_version: "encryptionkey@0"
I20260927 15:06:42.033624 2900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:42.038290 2915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:42.039994 2900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:42.040202 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "4accfdb4ab2b4923aecd1030dbc3a608"
format_stamp: "Formatted at 2026-09-27 15:06:42 on dist-test-slave-g0w1"
server_key: "2bb67046ea4d7f3f06c01148950a6d8b"
server_key_iv: "f801e5d86e3f64bac63100c180466e73"
server_key_version: "encryptionkey@0"
I20260927 15:06:42.040452 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:42.071758 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:42.076354 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:42.076679 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:42.077720 2900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:42.079236 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:42.079337 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:42.079493 2900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:42.079571 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:42.115319 2900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41035
I20260927 15:06:42.115422 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41035 every 8 connection(s)
I20260927 15:06:42.116678 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:42.121341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 2900
I20260927 15:06:42.121589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:42.122206 388 external_mini_cluster.cc:1428] Setting key 019c5a6cc06755152cea3b62bf2047a1
Sep 27 15:06:42 dist-test-slave-g0w1 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:06:42.139829 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:42.119477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59347 (local address 127.0.97.62:38075)
0927 15:06:42.119928 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:42.119934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:42.121168 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 15:06:42.123925 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:42.123930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:42.123993 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:42.124125 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:42.125796 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.126624 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.128341 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.128756 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.128894 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:42.133322 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:42.133328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:42.133331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:42.133370 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:42.136389 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:42.137274 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:42.137278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:42.137281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:42.137404 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:42.137887 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:42.137891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:42.137894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:42.138163 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:42.138332 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:42.139126 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 15:06:42.139443 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260927 15:06:42.141052 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:42.120136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38075 (local address 127.0.97.1:59347)
0927 15:06:42.120976 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:42.121017 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:42.122947 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:42.124411 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:42.124421 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:42.124991 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:42.125606 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:42.125624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.126805 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.126809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:42.128171 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:42.128184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.129011 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.129019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:42.129275 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:42.130046 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:42.130083 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:42.133040 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:42.136580 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:42.136591 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:42.136619 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:42.137115 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:42.137526 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:42.137531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:42.137535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:42.137743 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:42.138299 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:42.138309 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:42.138694 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 15:06:42.138981 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 15:06:42.139731 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":304,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:06:42.143028 3028 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38075
I20260927 15:06:42.143426 3028 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:42.144495 3028 heartbeater.cc:511] Master 127.0.97.62:38075 requested a full tablet report, sending...
I20260927 15:06:42.147159 2839 ts_manager.cc:194] Registered new tserver with Master: 4accfdb4ab2b4923aecd1030dbc3a608 (127.0.97.1:41035)
I20260927 15:06:42.148705 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:42.166834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38075
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:42.532075 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:42.532397 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:42.532493 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:42.540786 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:42.541006 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:42.541193 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:42.552019 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:42.553398 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:42.554774 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:42.569555 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:42.569621 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:42.569929 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:42.571051 3034 server_base.cc:1034] running on GCE node
I20260927 15:06:42.571995 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:42.573294 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:42.574555 3034 hybrid_clock.cc:648] HybridClock initialized: now 1790521602574481 us; error 87 us; skew 500 ppm
I20260927 15:06:42.574968 3034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:42 dist-test-slave-g0w1 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:42.578379 3034 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:42.579982 3034 webserver.cc:460] Webserver started at http://127.0.97.2:35243/ using document root <none> and password file <none>
I20260927 15:06:42.580689 3034 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:42.580796 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:42.581084 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:42.584239 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "67ceb3c6ccd74167a0fb6ce76bc922b9"
format_stamp: "Formatted at 2026-09-27 15:06:42 on dist-test-slave-g0w1"
server_key: "85096197358374ec6b6cfb811d9852e4"
server_key_iv: "e7afe32bc22e96e2e101f0a022e36f87"
server_key_version: "encryptionkey@0"
I20260927 15:06:42.585107 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "67ceb3c6ccd74167a0fb6ce76bc922b9"
format_stamp: "Formatted at 2026-09-27 15:06:42 on dist-test-slave-g0w1"
server_key: "85096197358374ec6b6cfb811d9852e4"
server_key_iv: "e7afe32bc22e96e2e101f0a022e36f87"
server_key_version: "encryptionkey@0"
I20260927 15:06:42.590842 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 15:06:42.594695 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:42.596365 3034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:42.596560 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "67ceb3c6ccd74167a0fb6ce76bc922b9"
format_stamp: "Formatted at 2026-09-27 15:06:42 on dist-test-slave-g0w1"
server_key: "85096197358374ec6b6cfb811d9852e4"
server_key_iv: "e7afe32bc22e96e2e101f0a022e36f87"
server_key_version: "encryptionkey@0"
I20260927 15:06:42.596840 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:42.616796 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:42.621625 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:42.621953 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:42.623035 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:42.624518 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:42.624629 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:42.624794 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:42.624869 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:42.661275 3034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39309
I20260927 15:06:42.661377 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39309 every 8 connection(s)
I20260927 15:06:42.662636 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:42.668322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3034
I20260927 15:06:42.668514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:42.669039 388 external_mini_cluster.cc:1428] Setting key af234bbd1fa95ec64146d1ab37b278ce
Sep 27 15:06:42 dist-test-slave-g0w1 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:42.688263 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:42.665488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39815 (local address 127.0.97.62:38075)
0927 15:06:42.665974 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:42.665980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:42.667218 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 15:06:42.669426 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:42.669432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:42.669526 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:42.669678 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:42.672075 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.673426 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.675243 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.676079 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.676244 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:42.680437 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:42.680446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:42.680454 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:42.680505 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:42.684441 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:42.685391 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:42.685399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:42.685405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:42.685586 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:42.686067 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:42.686074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:42.686080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:42.686517 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:42.686709 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:42.687564 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 15:06:42.687799 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:06:42.689610 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:42.666142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38075 (local address 127.0.97.2:39815)
0927 15:06:42.666969 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:42.667013 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:42.669035 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:42.670218 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:42.670232 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:42.671149 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:42.671861 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:42.671880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.673602 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.673607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:42.674998 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:42.675011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:42.676319 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:42.676324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:42.676604 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:42.677382 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:42.677412 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:42.680108 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:42.684661 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:42.684675 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:42.684707 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:42.685218 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:42.685717 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:42.685721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:42.685723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:42.685928 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:42.686661 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:42.686667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:42.687170 (+ 503us) client_negotiation.cc:769] Sending connection context
0927 15:06:42.687467 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 15:06:42.688051 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260927 15:06:42.691421 3162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38075
I20260927 15:06:42.691756 3162 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:06:42.692572 3162 heartbeater.cc:511] Master 127.0.97.62:38075 requested a full tablet report, sending...
I20260927 15:06:42.694278 2839 ts_manager.cc:194] Registered new tserver with Master: 67ceb3c6ccd74167a0fb6ce76bc922b9 (127.0.97.2:39309)
I20260927 15:06:42.695210 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:42.719485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38075
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:43.069504 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:43.069818 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:43.069921 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:43.077446 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:43.077585 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:43.077700 3168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:43.087915 3168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:43.089305 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:43.090613 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:43.104843 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:43.104971 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:43.104807 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:43.106233 3168 server_base.cc:1034] running on GCE node
I20260927 15:06:43.106956 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:43.108016 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:43.109256 3168 hybrid_clock.cc:648] HybridClock initialized: now 1790521603109209 us; error 43 us; skew 500 ppm
I20260927 15:06:43.109647 3168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:43 dist-test-slave-g0w1 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:43.113065 3168 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:43.114784 3168 webserver.cc:460] Webserver started at http://127.0.97.3:41531/ using document root <none> and password file <none>
I20260927 15:06:43.115571 3168 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:43.115722 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:43.116046 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:43.118975 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "dacfac7b834745cbbc7dc705449b9d99"
format_stamp: "Formatted at 2026-09-27 15:06:43 on dist-test-slave-g0w1"
server_key: "562f4120e053bd051f6607809c10ec26"
server_key_iv: "9221dfc5dfee82adc3e459687cd3ad92"
server_key_version: "encryptionkey@0"
I20260927 15:06:43.119874 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dacfac7b834745cbbc7dc705449b9d99"
format_stamp: "Formatted at 2026-09-27 15:06:43 on dist-test-slave-g0w1"
server_key: "562f4120e053bd051f6607809c10ec26"
server_key_iv: "9221dfc5dfee82adc3e459687cd3ad92"
server_key_version: "encryptionkey@0"
I20260927 15:06:43.125530 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:06:43.129639 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:43.131266 3168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:43.131460 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "dacfac7b834745cbbc7dc705449b9d99"
format_stamp: "Formatted at 2026-09-27 15:06:43 on dist-test-slave-g0w1"
server_key: "562f4120e053bd051f6607809c10ec26"
server_key_iv: "9221dfc5dfee82adc3e459687cd3ad92"
server_key_version: "encryptionkey@0"
I20260927 15:06:43.131680 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:43.152084 3028 heartbeater.cc:503] Master 127.0.97.62:38075 was elected leader, sending a full tablet report...
I20260927 15:06:43.160105 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:43.165411 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:43.165730 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:43.166740 3168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:43.168210 3168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:43.168323 3168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:43.168485 3168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:43.168557 3168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:43.203681 3168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45275
I20260927 15:06:43.203747 3295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45275 every 8 connection(s)
I20260927 15:06:43.205163 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:43.213848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3168
I20260927 15:06:43.214066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:43.214483 388 external_mini_cluster.cc:1428] Setting key 7c056b0aca79972f354c2daab63ac60c
Sep 27 15:06:43 dist-test-slave-g0w1 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:43.226456 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:43.207964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39015 (local address 127.0.97.62:38075)
0927 15:06:43.208391 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:43.208396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:43.209529 (+ 1133us) server_negotiation.cc:407] Connection header received
0927 15:06:43.211471 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:43.211477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:43.211581 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:43.211714 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:43.213330 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.214196 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.216180 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.216744 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.216896 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:43.220585 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:43.220591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:43.220594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:43.220628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:43.223560 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:43.224299 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:43.224302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:43.224305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:43.224422 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:43.224801 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:43.224804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:43.224806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:43.225072 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:43.225181 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:43.226000 (+ 819us) server_negotiation.cc:299] Negotiation successful
0927 15:06:43.226180 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260927 15:06:43.227645 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:43.208602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38075 (local address 127.0.97.3:39015)
0927 15:06:43.209361 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:43.209401 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:43.211167 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:43.211986 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:43.211995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:43.212551 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:43.213141 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:43.213160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.214382 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.214387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:43.216023 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:43.216041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.216993 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.217000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:43.217309 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:43.218098 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:43.218121 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:43.220351 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:43.223701 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:43.223709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:43.223723 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:43.224175 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:43.224515 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:43.224518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:43.224520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:43.224702 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:43.225228 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:43.225236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:43.225585 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 15:06:43.225907 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 15:06:43.226332 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":277,"threads_started":1}
I20260927 15:06:43.229393 3296 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38075
I20260927 15:06:43.229830 3296 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:43.230598 3296 heartbeater.cc:511] Master 127.0.97.62:38075 requested a full tablet report, sending...
I20260927 15:06:43.232307 2838 ts_manager.cc:194] Registered new tserver with Master: dacfac7b834745cbbc7dc705449b9d99 (127.0.97.3:45275)
I20260927 15:06:43.233305 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39015
I20260927 15:06:43.243150 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:06:43.274183 3298 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:43.271703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.0.97.62:38075)
0927 15:06:43.271894 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:43.271899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:43.272482 (+ 583us) server_negotiation.cc:407] Connection header received
0927 15:06:43.272611 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:43.272616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:43.272682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:43.272780 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:43.273992 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60136: BlockingRecv error: recv got EOF from 127.0.0.1:60136 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:06:43.277431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2900
W20260927 15:06:43.280728 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2900: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:41035: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:06:43.280937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 2900
I20260927 15:06:43.296931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3034
W20260927 15:06:43.299769 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3034: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39309: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:06:43.299997 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3034
I20260927 15:06:43.315629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3168
W20260927 15:06:43.318873 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3168: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45275: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:06:43.319002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3168
I20260927 15:06:43.334547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 2807
I20260927 15:06:43.335852 2869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:43.439395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 2807
2026-09-27T15:06:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3747 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:43 dist-test-slave-g0w1 krb5kdc[3314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:43 dist-test-slave-g0w1 krb5kdc[3314](info): set up 2 sockets
Sep 27 15:06:43 dist-test-slave-g0w1 krb5kdc[3314](info): commencing operation
krb5kdc: starting...
W20260927 15:06:44.640668 388 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 15:06:44.795691 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.155s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 15:06:45.113520 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.318s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 15:06:45.338055 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.224s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:45 dist-test-slave-g0w1 krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:45.458673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:45.791189 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:45.791508 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:45.791667 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:45.799127 3327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:45.799247 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:45.799338 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:45.799432 3327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:45.799515 3327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:45.809815 3327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:45.811250 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:45.812655 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:45.824903 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:45.824908 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:45.824945 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:45.825922 3327 server_base.cc:1034] running on GCE node
I20260927 15:06:45.827026 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:45.828802 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:45.830052 3327 hybrid_clock.cc:648] HybridClock initialized: now 1790521605829988 us; error 63 us; skew 500 ppm
I20260927 15:06:45.830439 3327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:45 dist-test-slave-g0w1 krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:45.841164 3327 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:45.842957 3327 webserver.cc:460] Webserver started at http://127.0.97.62:39171/ using document root <none> and password file <none>
I20260927 15:06:45.843757 3327 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:45.843914 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:45.844249 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:45.847546 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "5e71c93e91d64dc2bc068a4f17d7d953"
format_stamp: "Formatted at 2026-09-27 15:06:45 on dist-test-slave-g0w1"
server_key: "6e7d2c38ccd929efc03a9f2620f04e4c"
server_key_iv: "a895e0bf20f771eef0946d02771a65b1"
server_key_version: "encryptionkey@0"
I20260927 15:06:45.848387 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e71c93e91d64dc2bc068a4f17d7d953"
format_stamp: "Formatted at 2026-09-27 15:06:45 on dist-test-slave-g0w1"
server_key: "6e7d2c38ccd929efc03a9f2620f04e4c"
server_key_iv: "a895e0bf20f771eef0946d02771a65b1"
server_key_version: "encryptionkey@0"
I20260927 15:06:45.854058 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:45.857923 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:45.859434 3327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:45.859648 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "5e71c93e91d64dc2bc068a4f17d7d953"
format_stamp: "Formatted at 2026-09-27 15:06:45 on dist-test-slave-g0w1"
server_key: "6e7d2c38ccd929efc03a9f2620f04e4c"
server_key_iv: "a895e0bf20f771eef0946d02771a65b1"
server_key_version: "encryptionkey@0"
I20260927 15:06:45.859872 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:45.896445 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:45.901202 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:45.901530 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:45.921559 3327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41779
I20260927 15:06:45.921577 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41779 every 8 connection(s)
I20260927 15:06:45.923133 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:45.927172 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:45.928825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3327
I20260927 15:06:45.929057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:45.929826 388 external_mini_cluster.cc:1428] Setting key 44570612e6f303c5ea10b50c0ada6466
I20260927 15:06:45.943149 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953: Bootstrap starting.
Sep 27 15:06:45 dist-test-slave-g0w1 krb5kdc[3314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:45.947191 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:45.948397 3394 log.cc:824] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:45.952138 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953: No bootstrap required, opened a new log
I20260927 15:06:45.955466 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:45.932036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.0.97.62:41779)
0927 15:06:45.932951 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:45.932966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:45.933014 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:06:45.934632 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:45.934657 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:45.935218 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:45.935594 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:45.937503 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:45.938958 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:45.940939 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:45.941706 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:45.941849 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:45.946941 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:45.946951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:45.946968 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:45.947017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:45.950739 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:45.951670 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:45.951679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:45.951687 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:45.951888 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:45.952341 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:45.952350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:45.952352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:45.952966 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:45.953293 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:45.953790 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 15:06:45.954262 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":442,"threads_started":1}
I20260927 15:06:45.961143 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:45.963600 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41779 } }
I20260927 15:06:45.964053 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:45.964161 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e71c93e91d64dc2bc068a4f17d7d953, State: Initialized, Role: FOLLOWER
I20260927 15:06:45.964789 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41779 } }
I20260927 15:06:45.965035 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:45.965167 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:45.965355 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:45.967935 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41779 } }
I20260927 15:06:45.968705 3394 leader_election.cc:302] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e71c93e91d64dc2bc068a4f17d7d953; no voters:
I20260927 15:06:45.969434 3394 leader_election.cc:288] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:45.969597 3400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:45.971395 3400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [term 1 LEADER]: Becoming Leader. State: Replica: 5e71c93e91d64dc2bc068a4f17d7d953, State: Running, Role: LEADER
I20260927 15:06:45.971938 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:45.972087 3400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41779 } }
I20260927 15:06:45.977041 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41779 } } }
I20260927 15:06:45.977483 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:45.978124 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e71c93e91d64dc2bc068a4f17d7d953. Latest consensus state: current_term: 1 leader_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e71c93e91d64dc2bc068a4f17d7d953" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41779 } } }
I20260927 15:06:45.978431 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:45.988462 3409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:45.994112 3409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:46.004987 3409 catalog_manager.cc:1347] Generated new cluster ID: fba4f7ade5bd4ca9bcba91925de4e082
I20260927 15:06:46.005164 3409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:46.016139 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:46.005351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.97.62:41779)
0927 15:06:46.005544 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:46.005551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:46.006621 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 15:06:46.006893 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:46.006899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:46.006992 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:46.007160 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:46.008388 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:46.009602 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:46.011542 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:46.012456 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:46.012604 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:46.013950 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:46.013959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:46.013964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:46.014881 (+ 917us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:46.014983 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:46.015025 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:46.015181 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:46.015394 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 15:06:46.015504 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260927 15:06:46.019678 3409 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:46.021590 3409 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:46.033566 3409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e71c93e91d64dc2bc068a4f17d7d953: Generated new TSK 0
I20260927 15:06:46.034739 3409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:46.052397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3327
I20260927 15:06:46.053751 3389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:46.145038 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3327
2026-09-27T15:06:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2700 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:46 dist-test-slave-g0w1 krb5kdc[3428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:46 dist-test-slave-g0w1 krb5kdc[3428](info): set up 2 sockets
Sep 27 15:06:46 dist-test-slave-g0w1 krb5kdc[3428](info): commencing operation
krb5kdc: starting...
W20260927 15:06:47.239645 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:47 dist-test-slave-g0w1 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:47.446934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40479
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:47.783166 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:47.783504 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:47.783594 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:47.791023 3441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:47.791224 3441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:47.791281 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:47.791338 3441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:47.791383 3441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:47.801663 3441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:47.802982 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:47.804268 3441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:47.816591 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:47.816529 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:47.816566 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:47.817732 3441 server_base.cc:1034] running on GCE node
I20260927 15:06:47.818714 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:47.820374 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:47.821651 3441 hybrid_clock.cc:648] HybridClock initialized: now 1790521607821590 us; error 50 us; skew 500 ppm
I20260927 15:06:47.822125 3441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:47 dist-test-slave-g0w1 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:47.825603 3441 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:47.827342 3441 webserver.cc:460] Webserver started at http://127.0.97.62:39629/ using document root <none> and password file <none>
I20260927 15:06:47.828085 3441 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:47.828190 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:47.828450 3441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:47.831714 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "72a82af4367146e198f4c563aa840789"
format_stamp: "Formatted at 2026-09-27 15:06:47 on dist-test-slave-g0w1"
server_key: "b07715e0f07aa420c488d476bea78106"
server_key_iv: "df0d44ffb9cdf13f316cef0ee39677bb"
server_key_version: "encryptionkey@0"
I20260927 15:06:47.832536 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "72a82af4367146e198f4c563aa840789"
format_stamp: "Formatted at 2026-09-27 15:06:47 on dist-test-slave-g0w1"
server_key: "b07715e0f07aa420c488d476bea78106"
server_key_iv: "df0d44ffb9cdf13f316cef0ee39677bb"
server_key_version: "encryptionkey@0"
I20260927 15:06:47.839802 3441 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 15:06:47.843911 3456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:47.845537 3441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:47.845772 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "72a82af4367146e198f4c563aa840789"
format_stamp: "Formatted at 2026-09-27 15:06:47 on dist-test-slave-g0w1"
server_key: "b07715e0f07aa420c488d476bea78106"
server_key_iv: "df0d44ffb9cdf13f316cef0ee39677bb"
server_key_version: "encryptionkey@0"
I20260927 15:06:47.845964 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:47.875327 3441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:47.880275 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:47.880633 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:47.900452 3441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40479
I20260927 15:06:47.900475 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40479 every 8 connection(s)
I20260927 15:06:47.901962 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:47.906198 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:47.906951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3441
I20260927 15:06:47.907200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:47.907809 388 external_mini_cluster.cc:1428] Setting key 9a5d3fcada508e0aeea2fe5c948dab2c
I20260927 15:06:47.920979 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789: Bootstrap starting.
Sep 27 15:06:47 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:47.925025 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:47.926505 3508 log.cc:824] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:47.930294 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789: No bootstrap required, opened a new log
I20260927 15:06:47.934394 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:47.910069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.0.97.62:40479)
0927 15:06:47.910939 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:47.910957 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:47.911012 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:06:47.913009 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:47.913043 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:47.913656 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:47.914160 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:47.915606 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:47.917648 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:47.919030 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:47.920160 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:47.920352 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:47.925215 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:47.925226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:47.925248 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:47.925298 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:47.929276 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:47.930127 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:47.930136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:47.930146 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:47.930381 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:47.930873 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:47.930880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:47.930888 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:47.931658 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:47.931970 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:47.932392 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 15:06:47.932904 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":333,"threads_started":1}
I20260927 15:06:47.940289 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a82af4367146e198f4c563aa840789" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40479 } }
I20260927 15:06:47.940789 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:47.940886 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a82af4367146e198f4c563aa840789, State: Initialized, Role: FOLLOWER
I20260927 15:06:47.941426 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a82af4367146e198f4c563aa840789" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40479 } }
I20260927 15:06:47.941628 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:47.941727 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:47.941890 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:47.944381 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a82af4367146e198f4c563aa840789" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40479 } }
I20260927 15:06:47.944938 3508 leader_election.cc:302] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a82af4367146e198f4c563aa840789; no voters:
I20260927 15:06:47.945571 3508 leader_election.cc:288] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:47.945964 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:47.947053 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [term 1 LEADER]: Becoming Leader. State: Replica: 72a82af4367146e198f4c563aa840789, State: Running, Role: LEADER
I20260927 15:06:47.947656 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a82af4367146e198f4c563aa840789" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40479 } }
I20260927 15:06:47.948108 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:47.951267 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a82af4367146e198f4c563aa840789. Latest consensus state: current_term: 1 leader_uuid: "72a82af4367146e198f4c563aa840789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a82af4367146e198f4c563aa840789" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40479 } } }
I20260927 15:06:47.951668 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:47.952135 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72a82af4367146e198f4c563aa840789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a82af4367146e198f4c563aa840789" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40479 } } }
I20260927 15:06:47.952531 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:47.954203 3521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:47.965104 3521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:47.975390 3521 catalog_manager.cc:1347] Generated new cluster ID: 84bfd1c2d2054240a2a550c9654fa0e5
I20260927 15:06:47.975556 3521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:48.018076 3521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:48.019586 3521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:48.033138 3521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72a82af4367146e198f4c563aa840789: Generated new TSK 0
I20260927 15:06:48.034150 3521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:48.083524 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40479
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:48.426796 3534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:48.427183 3534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:48.427297 3534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:48.434792 3534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:48.435024 3534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:48.435264 3534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:48.445479 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:48.446759 3534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:48.448213 3534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:48.462102 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:48.462189 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:48.462154 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:48.463709 3534 server_base.cc:1034] running on GCE node
I20260927 15:06:48.464489 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:48.465502 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:48.466732 3534 hybrid_clock.cc:648] HybridClock initialized: now 1790521608466675 us; error 46 us; skew 500 ppm
I20260927 15:06:48.467051 3534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:48 dist-test-slave-g0w1 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:48.470736 3534 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:48.472453 3534 webserver.cc:460] Webserver started at http://127.0.97.1:40457/ using document root <none> and password file <none>
I20260927 15:06:48.473224 3534 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:48.473359 3534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:48.473682 3534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:48.476799 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "48121fdaa41c4bc5a48c9bd22d61b455"
format_stamp: "Formatted at 2026-09-27 15:06:48 on dist-test-slave-g0w1"
server_key: "ae8f5c26303507cd3cd246567d1eea18"
server_key_iv: "a0b24d611dc05830ceef3805edf85660"
server_key_version: "encryptionkey@0"
I20260927 15:06:48.477655 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "48121fdaa41c4bc5a48c9bd22d61b455"
format_stamp: "Formatted at 2026-09-27 15:06:48 on dist-test-slave-g0w1"
server_key: "ae8f5c26303507cd3cd246567d1eea18"
server_key_iv: "a0b24d611dc05830ceef3805edf85660"
server_key_version: "encryptionkey@0"
I20260927 15:06:48.483412 3534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:06:48.487437 3549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:48.489157 3534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:48.489378 3534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "48121fdaa41c4bc5a48c9bd22d61b455"
format_stamp: "Formatted at 2026-09-27 15:06:48 on dist-test-slave-g0w1"
server_key: "ae8f5c26303507cd3cd246567d1eea18"
server_key_iv: "a0b24d611dc05830ceef3805edf85660"
server_key_version: "encryptionkey@0"
I20260927 15:06:48.489593 3534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:48.508945 3534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:48.513652 3534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:48.513981 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:48.515053 3534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:48.516619 3534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:48.516703 3534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:48.516834 3534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:48.516897 3534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:48.552029 3534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38657
I20260927 15:06:48.552134 3661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38657 every 8 connection(s)
I20260927 15:06:48.553385 3534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:48.563606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3534
I20260927 15:06:48.563859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:48.564615 388 external_mini_cluster.cc:1428] Setting key 84a5760c1a1f2de716f86c7c5734c032
Sep 27 15:06:48 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:48.575826 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:48.556152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38747 (local address 127.0.97.62:40479)
0927 15:06:48.556629 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:48.556634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:48.557821 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 15:06:48.559984 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:48.559990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:48.560086 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:48.560223 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:48.561856 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:48.562598 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:48.565115 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:48.565576 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:48.565694 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:48.569661 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:48.569667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:48.569670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:48.569705 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:48.572473 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:48.573354 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:48.573360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:48.573362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:48.573489 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:48.573959 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:48.573963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:48.573965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:48.574234 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:48.574392 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:48.575206 (+ 814us) server_negotiation.cc:299] Negotiation successful
0927 15:06:48.575459 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":225,"threads_started":1}
I20260927 15:06:48.576963 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:48.556779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40479 (local address 127.0.97.1:38747)
0927 15:06:48.557610 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:48.557666 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:48.559530 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:48.560488 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:48.560498 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:48.561040 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:48.561661 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:48.561679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:48.562781 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:48.562785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:48.564933 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:48.564950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:48.565816 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:48.565821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:48.566033 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:48.566772 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:48.566798 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:48.569377 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:48.572659 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:48.572670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:48.572691 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:48.573189 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:48.573609 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:48.573614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:48.573618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:48.573823 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:48.574375 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:48.574385 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:48.574764 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 15:06:48.575064 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 15:06:48.575690 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":284,"threads_started":1}
I20260927 15:06:48.578814 3662 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40479
I20260927 15:06:48.579260 3662 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:48.580090 3662 heartbeater.cc:511] Master 127.0.97.62:40479 requested a full tablet report, sending...
I20260927 15:06:48.583195 3473 ts_manager.cc:194] Registered new tserver with Master: 48121fdaa41c4bc5a48c9bd22d61b455 (127.0.97.1:38657)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:06:48.584980 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:48.613014 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40479
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:48.959291 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:48.959635 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:48.959806 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:48.967505 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:48.967676 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:48.967862 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:48.978546 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:48.980116 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:48.981621 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:48.996171 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:48.996109 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:48.996392 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:48.997211 3668 server_base.cc:1034] running on GCE node
I20260927 15:06:48.997870 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:48.998975 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:49.000218 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790521609000145 us; error 69 us; skew 500 ppm
I20260927 15:06:49.000567 3668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:49.003999 3668 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:49.005604 3668 webserver.cc:460] Webserver started at http://127.0.97.2:43959/ using document root <none> and password file <none>
I20260927 15:06:49.006302 3668 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:49.006407 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:49.006649 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:49.009815 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "04ca0c738bac4e09b408fbcd01035262"
format_stamp: "Formatted at 2026-09-27 15:06:49 on dist-test-slave-g0w1"
server_key: "61eb2e0b2bc84d3c076a95ab66317a13"
server_key_iv: "cb45cf524f355e66b2580deb3d8f7ee5"
server_key_version: "encryptionkey@0"
I20260927 15:06:49.010619 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "04ca0c738bac4e09b408fbcd01035262"
format_stamp: "Formatted at 2026-09-27 15:06:49 on dist-test-slave-g0w1"
server_key: "61eb2e0b2bc84d3c076a95ab66317a13"
server_key_iv: "cb45cf524f355e66b2580deb3d8f7ee5"
server_key_version: "encryptionkey@0"
I20260927 15:06:49.016511 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:06:49.020274 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:49.021832 3668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:49.022050 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "04ca0c738bac4e09b408fbcd01035262"
format_stamp: "Formatted at 2026-09-27 15:06:49 on dist-test-slave-g0w1"
server_key: "61eb2e0b2bc84d3c076a95ab66317a13"
server_key_iv: "cb45cf524f355e66b2580deb3d8f7ee5"
server_key_version: "encryptionkey@0"
I20260927 15:06:49.022255 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:49.047785 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:49.052991 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:49.053313 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:49.054366 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:49.055968 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:49.056066 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:49.056205 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:49.056272 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:49.090840 3668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39551
I20260927 15:06:49.090965 3795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39551 every 8 connection(s)
I20260927 15:06:49.092232 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:49.095371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3668
I20260927 15:06:49.095608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:49.096266 388 external_mini_cluster.cc:1428] Setting key 4bc1042101e267162d40bf814c1b5039
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:06:49.116391 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.095207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33495 (local address 127.0.97.62:40479)
0927 15:06:49.095770 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:49.095779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:49.096937 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 15:06:49.099199 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:49.099204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:49.099271 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:49.099400 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:49.102024 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.102992 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.105292 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.105767 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.105922 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.110116 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:49.110121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:49.110124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:49.110159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:49.113085 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.113959 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.113965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.113968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.114107 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.114551 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.114560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.114562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.114831 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:49.114979 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:49.115779 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 15:06:49.116006 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":253,"threads_started":1}
I20260927 15:06:49.117587 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.095829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40479 (local address 127.0.97.2:33495)
0927 15:06:49.096738 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:49.096802 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:49.098818 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:49.099704 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:49.099721 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:49.100644 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:49.101706 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:49.101739 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.103221 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.103230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:49.105110 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:49.105126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.106025 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.106032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:49.106298 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.107227 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:49.107272 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:49.109836 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:49.113267 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:49.113277 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:49.113302 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:49.113821 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:49.114219 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:49.114226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:49.114231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:49.114429 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:49.114974 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:49.114984 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:49.115397 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 15:06:49.115680 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 15:06:49.116265 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:06:49.119359 3796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40479
I20260927 15:06:49.119693 3796 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:49.120550 3796 heartbeater.cc:511] Master 127.0.97.62:40479 requested a full tablet report, sending...
I20260927 15:06:49.122237 3473 ts_manager.cc:194] Registered new tserver with Master: 04ca0c738bac4e09b408fbcd01035262 (127.0.97.2:39551)
I20260927 15:06:49.123289 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:49.143381 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40479
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:49.492012 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:49.492324 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:49.492429 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:49.500154 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:49.500283 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:49.500409 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:49.510607 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:49.512068 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:49.513401 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:49.528635 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:49.528733 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:49.528704 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:49.529695 3802 server_base.cc:1034] running on GCE node
I20260927 15:06:49.530474 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:49.531641 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:49.532883 3802 hybrid_clock.cc:648] HybridClock initialized: now 1790521609532835 us; error 39 us; skew 500 ppm
I20260927 15:06:49.533289 3802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:49.536855 3802 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:49.538466 3802 webserver.cc:460] Webserver started at http://127.0.97.3:40137/ using document root <none> and password file <none>
I20260927 15:06:49.539281 3802 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:49.539436 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:49.539757 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:49.542831 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "35cb55a40f7042d5ae3460379f8d568a"
format_stamp: "Formatted at 2026-09-27 15:06:49 on dist-test-slave-g0w1"
server_key: "e2dbfe2748e1f38490fc6c05c0f6d631"
server_key_iv: "5b4d09059e163886b31cb414f2016d1e"
server_key_version: "encryptionkey@0"
I20260927 15:06:49.543717 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "35cb55a40f7042d5ae3460379f8d568a"
format_stamp: "Formatted at 2026-09-27 15:06:49 on dist-test-slave-g0w1"
server_key: "e2dbfe2748e1f38490fc6c05c0f6d631"
server_key_iv: "5b4d09059e163886b31cb414f2016d1e"
server_key_version: "encryptionkey@0"
I20260927 15:06:49.549664 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:49.553768 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:49.555694 3802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:06:49.555897 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "35cb55a40f7042d5ae3460379f8d568a"
format_stamp: "Formatted at 2026-09-27 15:06:49 on dist-test-slave-g0w1"
server_key: "e2dbfe2748e1f38490fc6c05c0f6d631"
server_key_iv: "5b4d09059e163886b31cb414f2016d1e"
server_key_version: "encryptionkey@0"
I20260927 15:06:49.556115 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:49.589309 3662 heartbeater.cc:503] Master 127.0.97.62:40479 was elected leader, sending a full tablet report...
I20260927 15:06:49.609026 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:49.613991 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:49.614372 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:49.615522 3802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:49.617089 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:49.617168 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:49.617259 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:49.617303 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:49.652588 3802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36885
I20260927 15:06:49.652680 3929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36885 every 8 connection(s)
I20260927 15:06:49.653918 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:49.655155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3802
I20260927 15:06:49.655387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:49.656016 388 external_mini_cluster.cc:1428] Setting key c8f1d40d62cbd9aebad6462feadcfc1b
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:49.676101 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.656790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50183 (local address 127.0.97.62:40479)
0927 15:06:49.657274 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:49.657280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:49.658365 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 15:06:49.660536 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:49.660541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:49.660632 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:49.660741 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:49.662396 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.663201 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.664785 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.665192 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.665323 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.669398 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:49.669404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:49.669407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:49.669443 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:49.672833 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.673622 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.673627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.673629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.673811 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.674317 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.674321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.674324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.674606 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:49.674729 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:49.675567 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 15:06:49.675756 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260927 15:06:49.677088 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.657309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40479 (local address 127.0.97.3:50183)
0927 15:06:49.658204 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:49.658256 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:49.660235 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:49.660996 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:49.661005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:49.661612 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:49.662229 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:49.662247 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.663363 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.663372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:49.664633 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:49.664646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.665449 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.665456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:49.665706 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.666439 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:49.666464 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:49.669091 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:49.672973 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:49.672981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:49.672996 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:49.673498 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:49.673971 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:49.673975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:49.673977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:49.674179 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:49.674724 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:49.674730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:49.675192 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 15:06:49.675461 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 15:06:49.675922 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":329,"threads_started":1}
I20260927 15:06:49.678465 3930 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40479
I20260927 15:06:49.678787 3930 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:49.679615 3930 heartbeater.cc:511] Master 127.0.97.62:40479 requested a full tablet report, sending...
I20260927 15:06:49.681272 3473 ts_manager.cc:194] Registered new tserver with Master: 35cb55a40f7042d5ae3460379f8d568a (127.0.97.3:36885)
I20260927 15:06:49.682147 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50183
I20260927 15:06:49.685330 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:49.734915 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.720056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.97.62:40479)
0927 15:06:49.720237 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:49.720242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:49.720667 (+ 425us) server_negotiation.cc:407] Connection header received
0927 15:06:49.720817 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:49.720821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:49.720889 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:49.720989 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:49.722596 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.723432 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.724858 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.725464 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.725575 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.729918 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:49.729924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:49.729927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:49.729967 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:49.732586 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.733241 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.733245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.733248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.733379 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.733706 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.733709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.733710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.734061 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:49.734240 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:49.734470 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:06:49.734648 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:06:49.735517 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38760
I20260927 15:06:49.737599 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3534
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:49.754297 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.738702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.0.97.1:38657)
0927 15:06:49.739313 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:49.739321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:49.739343 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:06:49.739401 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:49.739408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:49.739619 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:49.739779 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:49.740974 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.742027 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.743283 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.743853 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.744005 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.748111 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:49.748125 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:49.748144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:49.748185 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:49.751432 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.752173 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.752181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.752183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.752308 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.752698 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.752703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.752705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.753056 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:49.753371 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:49.753663 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:06:49.753932 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":293,"threads_started":1}
W20260927 15:06:49.755775 3657 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45524
W20260927 15:06:49.757012 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:49.757282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3534
I20260927 15:06:49.774454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3668
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:49.792294 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.775464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.0.97.2:39551)
0927 15:06:49.776059 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:49.776067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:49.776094 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:06:49.776153 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:49.776160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:49.776386 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:49.776521 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:49.777854 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.778934 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.780412 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.781126 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.781291 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.785856 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:49.785872 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:49.785888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:49.785942 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:49.789183 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.790034 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.790042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.790045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.790188 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.790623 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.790627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.790629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.791130 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:49.791402 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:49.791737 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:06:49.791971 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":308,"threads_started":1}
W20260927 15:06:49.793691 3791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59086
W20260927 15:06:49.794879 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:49.795125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3668
I20260927 15:06:49.810732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3802
Sep 27 15:06:49 dist-test-slave-g0w1 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:49.828943 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:49.811945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.0.97.3:36885)
0927 15:06:49.812581 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:49.812588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:49.812621 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:06:49.812695 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:49.812701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:49.812961 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:49.813184 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:49.814635 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.815794 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.817103 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:49.817695 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:49.817855 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:49.822631 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:49.822648 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:49.822666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:49.822708 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:49.825720 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.826523 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.826531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.826533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.826666 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:49.827158 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:49.827165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:49.827173 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:49.827620 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:49.827990 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:49.828337 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 15:06:49.828568 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":360,"threads_started":1}
W20260927 15:06:49.830320 3925 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46852
W20260927 15:06:49.831799 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3802: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:06:49.831952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3802
I20260927 15:06:49.848446 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3441
I20260927 15:06:49.849785 3503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:49.955142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3441
2026-09-27T15:06:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3819 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:50 dist-test-slave-g0w1 krb5kdc[3948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:50 dist-test-slave-g0w1 krb5kdc[3948](info): set up 2 sockets
Sep 27 15:06:50 dist-test-slave-g0w1 krb5kdc[3948](info): commencing operation
krb5kdc: starting...
W20260927 15:06:51.051721 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:51 dist-test-slave-g0w1 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:51.254777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:51.594806 3961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:51.595244 3961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:51.595420 3961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:51.605870 3961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:51.606045 3961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:51.606172 3961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:51.606248 3961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:51.606328 3961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:51.617705 3961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:51.619242 3961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:51.620683 3961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:51.632555 3966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:51.633101 3967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:51.633934 3969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:51.635165 3961 server_base.cc:1034] running on GCE node
I20260927 15:06:51.636034 3961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:51.637627 3961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:51.638897 3961 hybrid_clock.cc:648] HybridClock initialized: now 1790521611638812 us; error 36 us; skew 500 ppm
I20260927 15:06:51.639380 3961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:51 dist-test-slave-g0w1 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:51.651358 3961 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:51.653522 3961 webserver.cc:460] Webserver started at http://127.0.97.62:46241/ using document root <none> and password file <none>
I20260927 15:06:51.654395 3961 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:51.654582 3961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:51.654953 3961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:51.658463 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "9616a227b2da4b5f9413f85e1de4abe2"
format_stamp: "Formatted at 2026-09-27 15:06:51 on dist-test-slave-g0w1"
server_key: "e708cbc79c988e2ae8dadffba3b1e879"
server_key_iv: "03ad57e8d71b3ed4802ce648276e67a2"
server_key_version: "encryptionkey@0"
I20260927 15:06:51.659477 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "9616a227b2da4b5f9413f85e1de4abe2"
format_stamp: "Formatted at 2026-09-27 15:06:51 on dist-test-slave-g0w1"
server_key: "e708cbc79c988e2ae8dadffba3b1e879"
server_key_iv: "03ad57e8d71b3ed4802ce648276e67a2"
server_key_version: "encryptionkey@0"
I20260927 15:06:51.665803 3961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 15:06:51.670151 3976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:51.671918 3961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:06:51.672163 3961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "9616a227b2da4b5f9413f85e1de4abe2"
format_stamp: "Formatted at 2026-09-27 15:06:51 on dist-test-slave-g0w1"
server_key: "e708cbc79c988e2ae8dadffba3b1e879"
server_key_iv: "03ad57e8d71b3ed4802ce648276e67a2"
server_key_version: "encryptionkey@0"
I20260927 15:06:51.672403 3961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:51.693382 3961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:51.698330 3961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:51.698715 3961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:51.720253 3961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42277
I20260927 15:06:51.720284 4027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42277 every 8 connection(s)
I20260927 15:06:51.721808 3961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:51.724886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 3961
I20260927 15:06:51.725126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:51.725759 388 external_mini_cluster.cc:1428] Setting key cd22e1edb6b2a400c2f0f5d1899bc253
I20260927 15:06:51.726166 4028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:51.742020 4028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2: Bootstrap starting.
Sep 27 15:06:51 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:51.746230 4028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:51.747663 4028 log.cc:824] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:51.750752 4028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2: No bootstrap required, opened a new log
I20260927 15:06:51.752226 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:51.727873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.0.97.62:42277)
0927 15:06:51.728693 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.728706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.728761 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:06:51.731392 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.731423 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.731934 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.732381 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.734419 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.735903 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.737746 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.738555 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.738716 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:51.743551 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:51.743564 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:51.743588 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:51.743654 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:51.747747 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.748593 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.748604 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.748612 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.748846 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.749291 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.749295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.749297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.749912 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:51.750215 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:51.750678 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 15:06:51.751198 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":402,"threads_started":1}
I20260927 15:06:51.759328 4028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9616a227b2da4b5f9413f85e1de4abe2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42277 } }
I20260927 15:06:51.759754 4028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:51.759879 4028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9616a227b2da4b5f9413f85e1de4abe2, State: Initialized, Role: FOLLOWER
I20260927 15:06:51.760402 4028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9616a227b2da4b5f9413f85e1de4abe2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42277 } }
I20260927 15:06:51.760591 4028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:51.760725 4028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:51.760861 4028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:51.763491 4028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9616a227b2da4b5f9413f85e1de4abe2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42277 } }
I20260927 15:06:51.763998 4028 leader_election.cc:302] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9616a227b2da4b5f9413f85e1de4abe2; no voters:
I20260927 15:06:51.764544 4028 leader_election.cc:288] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:51.764700 4033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:51.765477 4033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [term 1 LEADER]: Becoming Leader. State: Replica: 9616a227b2da4b5f9413f85e1de4abe2, State: Running, Role: LEADER
I20260927 15:06:51.766044 4033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9616a227b2da4b5f9413f85e1de4abe2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42277 } }
I20260927 15:06:51.766776 4028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:51.768765 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9616a227b2da4b5f9413f85e1de4abe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9616a227b2da4b5f9413f85e1de4abe2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42277 } } }
I20260927 15:06:51.769312 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:51.769186 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9616a227b2da4b5f9413f85e1de4abe2. Latest consensus state: current_term: 1 leader_uuid: "9616a227b2da4b5f9413f85e1de4abe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9616a227b2da4b5f9413f85e1de4abe2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42277 } } }
I20260927 15:06:51.769482 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:51.771759 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:51.776911 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:51.787243 4038 catalog_manager.cc:1347] Generated new cluster ID: 999e230fe2794d91a43ad022c797d50d
I20260927 15:06:51.787408 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:51.814064 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:51.815573 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:51.830582 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9616a227b2da4b5f9413f85e1de4abe2: Generated new TSK 0
I20260927 15:06:51.831566 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:51.883632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42277
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:52.234114 4054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:52.234433 4054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:52.234591 4054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:52.242162 4054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:52.242293 4054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:52.242487 4054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:52.252895 4054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:52.254245 4054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:52.255678 4054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:52.268967 4059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:52.268929 4060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:52.269013 4062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:52.270408 4054 server_base.cc:1034] running on GCE node
I20260927 15:06:52.271190 4054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:52.272238 4054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:52.273514 4054 hybrid_clock.cc:648] HybridClock initialized: now 1790521612273463 us; error 39 us; skew 500 ppm
I20260927 15:06:52.273900 4054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:52 dist-test-slave-g0w1 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:52.277565 4054 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:52.279783 4054 webserver.cc:460] Webserver started at http://127.0.97.1:45127/ using document root <none> and password file <none>
I20260927 15:06:52.280567 4054 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:52.280697 4054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:52.281026 4054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:52.284118 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "f70cba923ec440a2b9606855e4b6faa4"
format_stamp: "Formatted at 2026-09-27 15:06:52 on dist-test-slave-g0w1"
server_key: "6c0e846dc11b4ecf7dc3fca7e8b45af2"
server_key_iv: "7a6e123f4fa7c9deab750a395cf8b826"
server_key_version: "encryptionkey@0"
I20260927 15:06:52.284992 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f70cba923ec440a2b9606855e4b6faa4"
format_stamp: "Formatted at 2026-09-27 15:06:52 on dist-test-slave-g0w1"
server_key: "6c0e846dc11b4ecf7dc3fca7e8b45af2"
server_key_iv: "7a6e123f4fa7c9deab750a395cf8b826"
server_key_version: "encryptionkey@0"
I20260927 15:06:52.290623 4054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:52.294399 4069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:52.296000 4054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:52.296154 4054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "f70cba923ec440a2b9606855e4b6faa4"
format_stamp: "Formatted at 2026-09-27 15:06:52 on dist-test-slave-g0w1"
server_key: "6c0e846dc11b4ecf7dc3fca7e8b45af2"
server_key_iv: "7a6e123f4fa7c9deab750a395cf8b826"
server_key_version: "encryptionkey@0"
I20260927 15:06:52.296340 4054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:52.325574 4054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:52.330005 4054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:52.330322 4054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:52.331347 4054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:52.332684 4054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:52.332757 4054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:52.332921 4054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:52.332979 4054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:52.365854 4054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36149
I20260927 15:06:52.365911 4181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36149 every 8 connection(s)
I20260927 15:06:52.367385 4054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:52.377887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4054
I20260927 15:06:52.378100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:52.378742 388 external_mini_cluster.cc:1428] Setting key 4624ae47eb3164e557e9d68dc29e70d8
Sep 27 15:06:52 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:52.389076 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.370164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37943 (local address 127.0.97.62:42277)
0927 15:06:52.370623 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.370628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.371728 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 15:06:52.373653 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.373659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.373766 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.373886 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:52.375519 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.376222 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.377883 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.378432 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.378578 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.383138 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:52.383145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:52.383150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:52.383190 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:52.385812 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:52.386626 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:52.386632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:52.386640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:52.386760 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:52.387303 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:52.387308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:52.387311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:52.387576 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:52.387732 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.388468 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.388699 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260927 15:06:52.390123 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.370768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42277 (local address 127.0.97.1:37943)
0927 15:06:52.371559 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:52.371606 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:52.373365 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:52.374134 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:52.374143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:52.374664 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:52.375345 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:52.375365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.376388 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.376393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:52.377735 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:52.377748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.378661 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.378670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:52.378974 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.380055 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:52.380084 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:52.382854 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:52.385983 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:52.385994 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:52.386016 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:52.386488 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:52.386878 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:52.386883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:52.386887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:52.387161 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:52.387709 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:52.387717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:52.388085 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 15:06:52.388356 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 15:06:52.388965 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":359,"threads_started":1}
I20260927 15:06:52.391837 4182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42277
I20260927 15:06:52.392160 4182 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:52.392881 4182 heartbeater.cc:511] Master 127.0.97.62:42277 requested a full tablet report, sending...
I20260927 15:06:52.395423 3993 ts_manager.cc:194] Registered new tserver with Master: f70cba923ec440a2b9606855e4b6faa4 (127.0.97.1:36149)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:06:52.397675 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:52.424081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42277
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:52.762455 4188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:52.762765 4188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:52.762871 4188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:52.770486 4188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:52.770617 4188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:52.770731 4188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:52.781046 4188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:52.782312 4188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:52.783771 4188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:52.797501 4194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:52.797536 4193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:52.798365 4196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:52.798616 4188 server_base.cc:1034] running on GCE node
I20260927 15:06:52.799368 4188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:52.800338 4188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:52.801580 4188 hybrid_clock.cc:648] HybridClock initialized: now 1790521612801533 us; error 42 us; skew 500 ppm
I20260927 15:06:52.801901 4188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:52 dist-test-slave-g0w1 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:52.805133 4188 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:52.806633 4188 webserver.cc:460] Webserver started at http://127.0.97.2:33895/ using document root <none> and password file <none>
I20260927 15:06:52.807425 4188 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:52.807557 4188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:52.807862 4188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:52.810823 4188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "32c9277c1e834c649f086da0fefd82b7"
format_stamp: "Formatted at 2026-09-27 15:06:52 on dist-test-slave-g0w1"
server_key: "cb6a918669dd2a70ead97fa7c7b53b3f"
server_key_iv: "cc8339b44c27a7c72b026282b0e36b01"
server_key_version: "encryptionkey@0"
I20260927 15:06:52.811705 4188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "32c9277c1e834c649f086da0fefd82b7"
format_stamp: "Formatted at 2026-09-27 15:06:52 on dist-test-slave-g0w1"
server_key: "cb6a918669dd2a70ead97fa7c7b53b3f"
server_key_iv: "cc8339b44c27a7c72b026282b0e36b01"
server_key_version: "encryptionkey@0"
I20260927 15:06:52.817286 4188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:52.821380 4203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:52.822881 4188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:06:52.823079 4188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "32c9277c1e834c649f086da0fefd82b7"
format_stamp: "Formatted at 2026-09-27 15:06:52 on dist-test-slave-g0w1"
server_key: "cb6a918669dd2a70ead97fa7c7b53b3f"
server_key_iv: "cc8339b44c27a7c72b026282b0e36b01"
server_key_version: "encryptionkey@0"
I20260927 15:06:52.823352 4188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:52.840111 4188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:52.844843 4188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:52.845147 4188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:52.846182 4188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:52.847657 4188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:52.847738 4188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:52.847915 4188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:52.847978 4188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:52.885056 4188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37651
I20260927 15:06:52.885097 4315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37651 every 8 connection(s)
I20260927 15:06:52.886374 4188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:52.894763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4188
I20260927 15:06:52.894963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:52.895529 388 external_mini_cluster.cc:1428] Setting key e140bbac43f7005ac0f3558ded9f1115
Sep 27 15:06:52 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:52.909623 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.889039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52789 (local address 127.0.97.62:42277)
0927 15:06:52.889215 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.889219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.890500 (+ 1281us) server_negotiation.cc:407] Connection header received
0927 15:06:52.892343 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.892348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.892426 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.892545 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:52.894197 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.895302 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.897240 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.897690 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.897830 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.902436 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:52.902444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:52.902452 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:52.902498 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:52.905970 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:52.906852 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:52.906862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:52.906869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:52.907053 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:52.907512 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:52.907519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:52.907525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:52.907921 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:52.908068 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.908871 (+ 803us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.909144 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:06:52.910405 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.889566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42277 (local address 127.0.97.2:52789)
0927 15:06:52.890298 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:52.890342 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:52.892049 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:52.892823 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:52.892832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:52.893412 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:52.894012 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:52.894031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.895483 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.895488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:52.897076 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:52.897091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.897948 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.897956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:52.898211 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.898964 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:52.898986 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:52.902157 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:52.906144 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:52.906152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:52.906168 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:52.906628 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:52.907199 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:52.907202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:52.907205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:52.907396 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:52.908114 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:52.908121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:52.908477 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 15:06:52.908753 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 15:06:52.909184 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":305,"threads_started":1}
I20260927 15:06:52.912125 4316 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42277
I20260927 15:06:52.912511 4316 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:52.913552 4316 heartbeater.cc:511] Master 127.0.97.62:42277 requested a full tablet report, sending...
I20260927 15:06:52.915300 3993 ts_manager.cc:194] Registered new tserver with Master: 32c9277c1e834c649f086da0fefd82b7 (127.0.97.2:37651)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:06:52.916198 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:52.942165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42277
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:53.283572 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:53.283931 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:53.284096 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:53.291788 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:53.291967 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:53.292141 4321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:53.302641 4321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:53.304265 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:53.305790 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:53.319937 4326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:53.319937 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:53.320010 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:53.321188 4321 server_base.cc:1034] running on GCE node
I20260927 15:06:53.321914 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:53.323037 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:53.324331 4321 hybrid_clock.cc:648] HybridClock initialized: now 1790521613324266 us; error 50 us; skew 500 ppm
I20260927 15:06:53.324723 4321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:53.328567 4321 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:53.330313 4321 webserver.cc:460] Webserver started at http://127.0.97.3:40717/ using document root <none> and password file <none>
I20260927 15:06:53.331154 4321 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:53.331300 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:53.331637 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:53.334726 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "832ec28e005d4720a5096fa726bc6f98"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-g0w1"
server_key: "f0c4134fa7b9189fa99df98ffcb816e3"
server_key_iv: "7d7254e55e12e0ba39950281223e2917"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.335680 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "832ec28e005d4720a5096fa726bc6f98"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-g0w1"
server_key: "f0c4134fa7b9189fa99df98ffcb816e3"
server_key_iv: "7d7254e55e12e0ba39950281223e2917"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.341585 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:53.345744 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:53.347731 4321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:53.347985 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "832ec28e005d4720a5096fa726bc6f98"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-g0w1"
server_key: "f0c4134fa7b9189fa99df98ffcb816e3"
server_key_iv: "7d7254e55e12e0ba39950281223e2917"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.348237 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:53.375946 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:53.381062 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:53.381392 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:53.382449 4321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:53.383975 4321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:53.384089 4321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:53.384228 4321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:53.384296 4321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:53.401597 4182 heartbeater.cc:503] Master 127.0.97.62:42277 was elected leader, sending a full tablet report...
I20260927 15:06:53.420311 4321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37511
I20260927 15:06:53.420418 4448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37511 every 8 connection(s)
I20260927 15:06:53.421612 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:53.422932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4321
I20260927 15:06:53.423130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:53.423674 388 external_mini_cluster.cc:1428] Setting key daee39658d9332b583b7d3a5d6923cc9
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:53.444002 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.424620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54071 (local address 127.0.97.62:42277)
0927 15:06:53.425156 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.425163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.426918 (+ 1755us) server_negotiation.cc:407] Connection header received
0927 15:06:53.428710 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.428716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.428821 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.428946 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.430629 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.431670 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.433307 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.433771 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.433922 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.437702 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.437707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.437710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.437746 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.440765 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.441539 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.441545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.441549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.441675 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.442122 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.442130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.442133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.442421 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.442575 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.443380 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.443617 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":265,"threads_started":1}
I20260927 15:06:53.445086 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.425379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42277 (local address 127.0.97.3:54071)
0927 15:06:53.426366 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.426423 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:53.428415 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:53.429221 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:53.429230 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:53.429807 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:53.430426 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.430450 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.431844 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.431848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.433151 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.433163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.434033 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.434040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.434297 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.435014 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:53.435039 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:53.437436 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:53.440915 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.440926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.440946 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.441411 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.441797 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.441802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.441806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.442012 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.442552 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:53.442560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:53.442943 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 15:06:53.443280 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 15:06:53.443865 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":377,"threads_started":1}
I20260927 15:06:53.446619 4449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42277
I20260927 15:06:53.446998 4449 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:53.447770 4449 heartbeater.cc:511] Master 127.0.97.62:42277 requested a full tablet report, sending...
I20260927 15:06:53.449271 3993 ts_manager.cc:194] Registered new tserver with Master: 832ec28e005d4720a5096fa726bc6f98 (127.0.97.3:37511)
I20260927 15:06:53.450071 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54071
I20260927 15:06:53.452700 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:53.505616 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.488640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.0.97.1:36149)
0927 15:06:53.489410 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.489420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.489445 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:53.489504 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.489510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.489741 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.489934 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.491452 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.492513 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.493820 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.494423 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.494572 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.498853 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.498868 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.498885 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.498933 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.502322 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.503244 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.503252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.503255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.503404 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.504031 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.504035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.504038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.504415 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.504660 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.505077 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.505292 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":382,"threads_started":1}
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:53.525043 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.510460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35774 (local address 127.0.97.62:42277)
0927 15:06:53.510680 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.510686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.510900 (+ 214us) server_negotiation.cc:407] Connection header received
0927 15:06:53.511124 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.511130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.511211 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.511320 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.512648 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.513387 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.514644 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.515142 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.515280 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.519615 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.519621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.519625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.519669 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.522400 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.523183 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.523187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.523189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.523317 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.523647 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.523650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.523652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.523957 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.524146 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.524407 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.524643 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 15:06:53.525695 3993 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35774
I20260927 15:06:53.527035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4054
I20260927 15:06:53.539865 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.528004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.0.97.1:36149)
0927 15:06:53.528159 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.528164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.528634 (+ 470us) server_negotiation.cc:407] Connection header received
0927 15:06:53.528736 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.528740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.528895 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.528984 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.530223 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.530917 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.532172 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.532611 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.532711 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.534900 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.534905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.534908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.534943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.537426 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.538107 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.538111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.538113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.538228 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.538550 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.538553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.538555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.538798 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.538976 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.539455 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.539634 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:06:53.540560 4177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:53.624933 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4054
I20260927 15:06:53.642882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4188
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:53.660878 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.643911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.97.2:37651)
0927 15:06:53.644531 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.644539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.644567 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:06:53.644636 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.644643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.644868 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.645010 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.646306 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.647566 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.649279 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.649913 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.650060 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.654820 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.654837 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.654854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.654943 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.657877 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.658658 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.658666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.658669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.658796 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.659269 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.659273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.659275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.659626 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.659949 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.660252 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.660520 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
I20260927 15:06:53.662237 4311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:53.747466 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4188
I20260927 15:06:53.763368 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4321
Sep 27 15:06:53 dist-test-slave-g0w1 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:53.781028 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.764416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.0.97.3:37511)
0927 15:06:53.765086 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.765093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.765125 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:06:53.765187 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.765194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.765402 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.765555 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.766765 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.767961 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.769380 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.769970 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.770073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.774595 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.774624 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.774640 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.774681 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.777805 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.778572 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.778580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.778582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.778708 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.779377 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.779381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.779383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.779739 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.780063 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.780375 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.780676 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":349,"threads_started":1}
I20260927 15:06:53.782729 4444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:53.868064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4321
I20260927 15:06:53.883796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 3961
I20260927 15:06:53.885082 4023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:54.013134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 3961
2026-09-27T15:06:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4054 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:54 dist-test-slave-g0w1 krb5kdc[4469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:54 dist-test-slave-g0w1 krb5kdc[4469](info): set up 2 sockets
Sep 27 15:06:54 dist-test-slave-g0w1 krb5kdc[4469](info): commencing operation
krb5kdc: starting...
W20260927 15:06:55.139822 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:06:55 dist-test-slave-g0w1 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:06:55.358799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34719
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:55.698042 4482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:55.698365 4482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:55.698547 4482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:55.705971 4482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:55.706099 4482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:55.706187 4482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:55.706272 4482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:55.706372 4482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:55.716739 4482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:55.718159 4482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:55.719578 4482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:55.732034 4487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:55.732023 4488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:55.732072 4490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:55.733752 4482 server_base.cc:1034] running on GCE node
I20260927 15:06:55.734685 4482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:55.736326 4482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:55.737560 4482 hybrid_clock.cc:648] HybridClock initialized: now 1790521615737491 us; error 58 us; skew 500 ppm
I20260927 15:06:55.737977 4482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:55 dist-test-slave-g0w1 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:55.741513 4482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:06:55.742759 4482 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:55.742861 4482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:55.743227 4482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:55.746441 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "b78170d9b748444caaaf7413d2a62715"
format_stamp: "Formatted at 2026-09-27 15:06:55 on dist-test-slave-g0w1"
server_key: "2c6422fb2c27c927d55d30de15a16df1"
server_key_iv: "931451418a21014b5e4a3c5489893427"
server_key_version: "encryptionkey@0"
I20260927 15:06:55.747385 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "b78170d9b748444caaaf7413d2a62715"
format_stamp: "Formatted at 2026-09-27 15:06:55 on dist-test-slave-g0w1"
server_key: "2c6422fb2c27c927d55d30de15a16df1"
server_key_iv: "931451418a21014b5e4a3c5489893427"
server_key_version: "encryptionkey@0"
I20260927 15:06:55.753348 4482 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:55.757915 4496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:55.759931 4482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:06:55.760228 4482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "b78170d9b748444caaaf7413d2a62715"
format_stamp: "Formatted at 2026-09-27 15:06:55 on dist-test-slave-g0w1"
server_key: "2c6422fb2c27c927d55d30de15a16df1"
server_key_iv: "931451418a21014b5e4a3c5489893427"
server_key_version: "encryptionkey@0"
I20260927 15:06:55.760512 4482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:55.795550 4482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:55.800843 4482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:55.801191 4482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:55.822651 4482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34719
I20260927 15:06:55.822624 4547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34719 every 8 connection(s)
I20260927 15:06:55.823729 4482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:55.828181 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:55.830303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4482
I20260927 15:06:55.830547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:55.831262 388 external_mini_cluster.cc:1428] Setting key 064e08d1060de30dff771af43f8b47db
I20260927 15:06:55.842748 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: Bootstrap starting.
I20260927 15:06:55.847482 4548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: Neither blocks nor log segments found. Creating new log.
Sep 27 15:06:55 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:55.848855 4548 log.cc:824] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:55.852861 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: No bootstrap required, opened a new log
I20260927 15:06:55.859223 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:55.833471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.0.97.62:34719)
0927 15:06:55.834407 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:55.834422 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:55.834477 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:06:55.836320 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:55.836347 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:55.836834 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:55.837213 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:55.839500 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.840994 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.843225 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.844030 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.844187 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:55.849500 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:55.849517 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:55.849543 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:55.849606 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:55.853875 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:55.854775 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:55.854786 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:55.854811 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:55.855045 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:55.855546 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:55.855551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:55.855555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:55.856392 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:55.856720 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:55.857138 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 15:06:55.857646 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":431,"threads_started":1}
I20260927 15:06:55.863396 4548 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78170d9b748444caaaf7413d2a62715" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34719 } }
I20260927 15:06:55.863793 4548 raft_consensus.cc:374] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:55.863919 4548 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78170d9b748444caaaf7413d2a62715, State: Initialized, Role: FOLLOWER
I20260927 15:06:55.864571 4548 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78170d9b748444caaaf7413d2a62715" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34719 } }
I20260927 15:06:55.864789 4548 raft_consensus.cc:388] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:55.864934 4548 raft_consensus.cc:482] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:55.865119 4548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:55.868283 4548 raft_consensus.cc:504] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78170d9b748444caaaf7413d2a62715" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34719 } }
I20260927 15:06:55.869023 4548 leader_election.cc:302] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b78170d9b748444caaaf7413d2a62715; no voters:
I20260927 15:06:55.869910 4548 leader_election.cc:288] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:55.870070 4553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:55.871281 4553 raft_consensus.cc:686] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [term 1 LEADER]: Becoming Leader. State: Replica: b78170d9b748444caaaf7413d2a62715, State: Running, Role: LEADER
I20260927 15:06:55.871832 4553 consensus_queue.cc:237] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78170d9b748444caaaf7413d2a62715" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34719 } }
I20260927 15:06:55.872177 4548 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:55.874794 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b78170d9b748444caaaf7413d2a62715. Latest consensus state: current_term: 1 leader_uuid: "b78170d9b748444caaaf7413d2a62715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78170d9b748444caaaf7413d2a62715" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34719 } } }
I20260927 15:06:55.875324 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:55.874830 4554 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b78170d9b748444caaaf7413d2a62715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78170d9b748444caaaf7413d2a62715" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34719 } } }
I20260927 15:06:55.875494 4554 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715 [sys.catalog]: This master's current role is: LEADER
W20260927 15:06:55.886891 4569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:55.887135 4569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:55.887332 4559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:55.889056 4559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:55.897754 4559 catalog_manager.cc:1347] Generated new cluster ID: c9e769ccd1be4f6188fb6a987a39c03f
I20260927 15:06:55.897897 4559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:55.917407 4559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:55.918959 4559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:55.927349 4559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: Generated new TSK 0
I20260927 15:06:55.928267 4559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:06:55.984405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34719
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:56.336560 4574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:56.336879 4574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:56.337054 4574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:56.344655 4574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:56.344789 4574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:56.344970 4574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:56.355134 4574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:56.356554 4574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:56.357973 4574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:56.372771 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:56.372712 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:56.372686 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:56.373505 4574 server_base.cc:1034] running on GCE node
I20260927 15:06:56.374280 4574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:56.375435 4574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:56.376703 4574 hybrid_clock.cc:648] HybridClock initialized: now 1790521616376635 us; error 57 us; skew 500 ppm
I20260927 15:06:56.377112 4574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:56 dist-test-slave-g0w1 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:56.380736 4574 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:06:56.382067 4574 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:56.382167 4574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:56.382550 4574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:56.385767 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4"
format_stamp: "Formatted at 2026-09-27 15:06:56 on dist-test-slave-g0w1"
server_key: "fb17075f0624a1bc0570b31f38d506bc"
server_key_iv: "e140e976365b7565fe64f6f01272ab91"
server_key_version: "encryptionkey@0"
I20260927 15:06:56.386695 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4"
format_stamp: "Formatted at 2026-09-27 15:06:56 on dist-test-slave-g0w1"
server_key: "fb17075f0624a1bc0570b31f38d506bc"
server_key_iv: "e140e976365b7565fe64f6f01272ab91"
server_key_version: "encryptionkey@0"
I20260927 15:06:56.392657 4574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260927 15:06:56.396847 4588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:56.398622 4574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:06:56.398849 4574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4"
format_stamp: "Formatted at 2026-09-27 15:06:56 on dist-test-slave-g0w1"
server_key: "fb17075f0624a1bc0570b31f38d506bc"
server_key_iv: "e140e976365b7565fe64f6f01272ab91"
server_key_version: "encryptionkey@0"
I20260927 15:06:56.399384 4574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:56.430095 4574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:56.434701 4574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:56.435005 4574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:56.436062 4574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:56.437531 4574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:56.437619 4574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:56.437755 4574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:56.437831 4574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:56.472122 4574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43505
I20260927 15:06:56.472169 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43505 every 8 connection(s)
I20260927 15:06:56.473193 4574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:56.477804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4574
I20260927 15:06:56.478014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:56.478852 388 external_mini_cluster.cc:1428] Setting key d13d2d752c0e8b962f5a993512ff2c96
Sep 27 15:06:56 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:56.497206 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:56.476010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58573 (local address 127.0.97.62:34719)
0927 15:06:56.476416 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:56.476432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:56.477559 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 15:06:56.480013 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:56.480019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:56.480117 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:56.480254 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:56.482072 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:56.482798 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:56.484523 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:56.484933 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:56.485072 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:56.489818 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:56.489823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:56.489827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:56.489865 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:56.492978 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:56.494256 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:56.494263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:56.494267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:56.494499 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:56.495154 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:56.495159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:56.495161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:56.495498 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:56.495617 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:56.496716 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0927 15:06:56.496921 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":200,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:06:56.498787 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:56.476639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34719 (local address 127.0.97.1:58573)
0927 15:06:56.477360 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:56.477400 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:56.479689 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:56.480517 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:56.480526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:56.481254 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:56.481874 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:56.481893 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:56.482961 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:56.482965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:56.484329 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:56.484343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:56.485179 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:56.485187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:56.485428 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:56.486256 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:56.486300 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:56.489519 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:56.493214 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:56.493229 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:56.493253 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:56.494031 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:56.494638 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:56.494643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:56.494648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:56.494939 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:56.495644 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:56.495655 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:56.496224 (+ 569us) client_negotiation.cc:769] Sending connection context
0927 15:06:56.496580 (+ 356us) client_negotiation.cc:240] Negotiation successful
0927 15:06:56.497214 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":299,"threads_started":1}
I20260927 15:06:56.500837 4701 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34719
I20260927 15:06:56.501195 4701 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:06:56.502031 4701 heartbeater.cc:511] Master 127.0.97.62:34719 requested a full tablet report, sending...
I20260927 15:06:56.504740 4513 ts_manager.cc:194] Registered new tserver with Master: 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 (127.0.97.1:43505)
I20260927 15:06:56.506281 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:06:56.527012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34719
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:56.873883 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:56.874277 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:56.874446 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:56.882550 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:56.882678 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:56.882864 4707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:56.893335 4707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:56.894726 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:56.896230 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:56.910135 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:56.910383 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:56.910135 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:56.911484 4707 server_base.cc:1034] running on GCE node
I20260927 15:06:56.912338 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:56.913493 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:56.914768 4707 hybrid_clock.cc:648] HybridClock initialized: now 1790521616914716 us; error 46 us; skew 500 ppm
I20260927 15:06:56.915223 4707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:56 dist-test-slave-g0w1 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:56.925128 4707 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:06:56.926558 4707 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:56.926714 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:56.927127 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:56.930472 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "66b06bf4ae0f4a9389003a1209fb0c42"
format_stamp: "Formatted at 2026-09-27 15:06:56 on dist-test-slave-g0w1"
server_key: "3176c854b8b890199eddeebbdb2e989f"
server_key_iv: "77f00be9946b7a9a9c631b5c4f636855"
server_key_version: "encryptionkey@0"
I20260927 15:06:56.931533 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "66b06bf4ae0f4a9389003a1209fb0c42"
format_stamp: "Formatted at 2026-09-27 15:06:56 on dist-test-slave-g0w1"
server_key: "3176c854b8b890199eddeebbdb2e989f"
server_key_iv: "77f00be9946b7a9a9c631b5c4f636855"
server_key_version: "encryptionkey@0"
I20260927 15:06:56.937382 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 15:06:56.941619 4721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:56.943192 4707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:56.943395 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "66b06bf4ae0f4a9389003a1209fb0c42"
format_stamp: "Formatted at 2026-09-27 15:06:56 on dist-test-slave-g0w1"
server_key: "3176c854b8b890199eddeebbdb2e989f"
server_key_iv: "77f00be9946b7a9a9c631b5c4f636855"
server_key_version: "encryptionkey@0"
I20260927 15:06:56.943643 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:56.969497 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:56.974462 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:56.974769 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:56.975888 4707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:56.977319 4707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:56.977461 4707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:56.977608 4707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:56.977699 4707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:57.015798 4707 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37587
I20260927 15:06:57.015859 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37587 every 8 connection(s)
I20260927 15:06:57.016695 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:57.023265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4707
I20260927 15:06:57.023483 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:57.024011 388 external_mini_cluster.cc:1428] Setting key 1b5ce27e9292ba33b4f7c491f104b2b5
Sep 27 15:06:57 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:57.039325 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.019358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55213 (local address 127.0.97.62:34719)
0927 15:06:57.019801 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.019807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.020834 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 15:06:57.022642 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.022647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.022718 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.022840 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:57.025192 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.026098 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.027919 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.028736 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.028933 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.032971 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:57.032976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:57.032980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:57.033017 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:57.035921 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.036806 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.036811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.036813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.036945 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.037425 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.037429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.037431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.037706 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:57.037863 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.038632 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.038874 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260927 15:06:57.040508 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.019973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34719 (local address 127.0.97.2:55213)
0927 15:06:57.020659 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.020704 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.022358 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.023232 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.023249 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.024083 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:57.024948 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.024974 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.026253 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.026257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.027636 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.027650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.029043 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.029050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.029345 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.030184 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:57.030209 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:57.032659 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:57.036115 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:57.036126 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:57.036147 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:57.036642 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:57.037069 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:57.037074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:57.037078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:57.037285 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:57.037846 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:57.037856 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:57.038247 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.038540 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.039184 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":281,"threads_started":1}
I20260927 15:06:57.042248 4834 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34719
I20260927 15:06:57.042692 4834 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:06:57.043648 4834 heartbeater.cc:511] Master 127.0.97.62:34719 requested a full tablet report, sending...
I20260927 15:06:57.045634 4513 ts_manager.cc:194] Registered new tserver with Master: 66b06bf4ae0f4a9389003a1209fb0c42 (127.0.97.2:37587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:06:57.046586 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:06:57.070767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34719
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:57.411257 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:57.411574 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:57.411743 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:57.419278 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:57.419442 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:57.419638 4840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:06:57.431576 4840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:57.433110 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:57.434613 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:57.449831 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:57.449779 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:57.449779 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:57.451675 4840 server_base.cc:1034] running on GCE node
I20260927 15:06:57.452354 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:57.453437 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:57.454682 4840 hybrid_clock.cc:648] HybridClock initialized: now 1790521617454632 us; error 46 us; skew 500 ppm
I20260927 15:06:57.455035 4840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:06:57 dist-test-slave-g0w1 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:57.458525 4840 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:06:57.459885 4840 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:57.460047 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:57.460369 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:57.463598 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "ad63956a0c854bf287b2938de484b2a6"
format_stamp: "Formatted at 2026-09-27 15:06:57 on dist-test-slave-g0w1"
server_key: "5a8432db5a6a62b7d3c0f61493f1fafc"
server_key_iv: "3e8e64166aa67c3e322580b376554011"
server_key_version: "encryptionkey@0"
I20260927 15:06:57.464573 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ad63956a0c854bf287b2938de484b2a6"
format_stamp: "Formatted at 2026-09-27 15:06:57 on dist-test-slave-g0w1"
server_key: "5a8432db5a6a62b7d3c0f61493f1fafc"
server_key_iv: "3e8e64166aa67c3e322580b376554011"
server_key_version: "encryptionkey@0"
I20260927 15:06:57.470304 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:57.474750 4854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:57.476473 4840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 15:06:57.476694 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "ad63956a0c854bf287b2938de484b2a6"
format_stamp: "Formatted at 2026-09-27 15:06:57 on dist-test-slave-g0w1"
server_key: "5a8432db5a6a62b7d3c0f61493f1fafc"
server_key_iv: "3e8e64166aa67c3e322580b376554011"
server_key_version: "encryptionkey@0"
I20260927 15:06:57.476950 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:57.491914 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:57.496629 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:57.496948 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:57.498008 4840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:57.499596 4840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:57.499727 4840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:57.499866 4840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:57.499960 4840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:57.509820 4701 heartbeater.cc:503] Master 127.0.97.62:34719 was elected leader, sending a full tablet report...
I20260927 15:06:57.536863 4840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41941
I20260927 15:06:57.537101 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41941 every 8 connection(s)
I20260927 15:06:57.537909 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:57.541745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 4840
I20260927 15:06:57.541940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:57.542388 388 external_mini_cluster.cc:1428] Setting key 70ae18f17040489df9eadc3eb9dbd0d6
Sep 27 15:06:57 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:06:57.560639 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.540708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44145 (local address 127.0.97.62:34719)
0927 15:06:57.541251 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.541256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.542568 (+ 1312us) server_negotiation.cc:407] Connection header received
0927 15:06:57.545005 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.545010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.545121 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.545262 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:57.547017 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.548056 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.549820 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.550271 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.550379 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.554268 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:57.554274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:57.554277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:57.554314 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:57.557373 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.558163 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.558171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.558178 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.558309 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.558742 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.558749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.558752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.559052 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:57.559269 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.560014 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.560248 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":324,"threads_started":1}
I20260927 15:06:57.561734 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.541371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34719 (local address 127.0.97.3:44145)
0927 15:06:57.542303 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.542363 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.544675 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.545542 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.545552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.546131 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:57.546797 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.546841 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.548222 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.548226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.549672 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.549686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.550525 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.550531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.550764 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.551496 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:57.551525 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:57.554014 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:57.557544 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:57.557555 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:57.557582 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:57.558037 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:57.558433 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:57.558439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:57.558442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:57.558626 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:57.559242 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:57.559251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:57.559628 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.559927 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.560499 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260927 15:06:57.563597 4967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34719
I20260927 15:06:57.563946 4967 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:57.564697 4967 heartbeater.cc:511] Master 127.0.97.62:34719 requested a full tablet report, sending...
I20260927 15:06:57.566298 4513 ts_manager.cc:194] Registered new tserver with Master: ad63956a0c854bf287b2938de484b2a6 (127.0.97.3:41941)
I20260927 15:06:57.567245 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44145
I20260927 15:06:57.570573 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:57.589161 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.575331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.0.97.62:34719)
0927 15:06:57.575508 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.575513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.576008 (+ 495us) server_negotiation.cc:407] Connection header received
0927 15:06:57.576190 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.576194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.576261 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.576349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:57.578143 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.578871 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.580574 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.581136 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.581269 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.583777 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:57.583783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:57.583787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:57.583832 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:57.586621 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.587447 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.587451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.587453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.587588 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.587921 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.587925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.587928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.588245 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:57.588416 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.588726 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.588913 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:06:57.593451 4513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:57.600306 4513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:57.627581 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.613060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43505 (local address 127.0.0.1:56102)
0927 15:06:57.613938 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.613986 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.614216 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.616244 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.616257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.616302 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:57.616820 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.616838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.619806 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.619816 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.623232 (+ 3416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.623256 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.626041 (+ 2785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.626051 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.626358 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.626460 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.626613 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.626915 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":1,"thread_start_us":435,"threads_started":1}
I20260927 15:06:57.629630 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.613205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.0.97.1:43505)
0927 15:06:57.615416 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.615437 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.615468 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:06:57.615584 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.615593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.615922 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.617106 (+ 1184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:57.617391 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.619580 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.623556 (+ 3976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.625726 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.628731 (+ 3005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.629042 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.629120 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.629294 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2048,"thread_start_us":1872,"threads_started":1}
I20260927 15:06:57.631242 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.613137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41941 (local address 127.0.0.1:47094)
0927 15:06:57.615138 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.615160 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.616124 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.617352 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.617360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.617400 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:57.617964 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.617980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.624087 (+ 6107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.624095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.628578 (+ 4483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.628596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.630398 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.630407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.630590 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.630624 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.630712 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.630867 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1839,"thread_start_us":2083,"threads_started":1}
I20260927 15:06:57.631282 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.613686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.0.97.3:41941)
0927 15:06:57.614866 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.614874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.615358 (+ 484us) server_negotiation.cc:407] Connection header received
0927 15:06:57.616395 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.616408 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.616737 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.616995 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:57.619919 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.623872 (+ 3953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.628758 (+ 4886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.630197 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.630312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.630470 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.630786 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.630956 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":885,"threads_started":1}
I20260927 15:06:57.633821 4636 tablet_service.cc:1467] Processing CreateTablet for tablet e78616a2e9ac4cf686e438ed7878de48 (DEFAULT_TABLE table=test-table [id=efebcfee9c54438b9e54b10d1bd87827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:57.635322 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78616a2e9ac4cf686e438ed7878de48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:57.635140 4902 tablet_service.cc:1467] Processing CreateTablet for tablet e78616a2e9ac4cf686e438ed7878de48 (DEFAULT_TABLE table=test-table [id=efebcfee9c54438b9e54b10d1bd87827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:57.636520 4902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78616a2e9ac4cf686e438ed7878de48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:57.636543 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.615615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37587 (local address 127.0.0.1:49338)
0927 15:06:57.628348 (+ 12733us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.628379 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.628540 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.629285 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.629289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.629315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:57.629491 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.629504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.632124 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.632129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.633936 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.633960 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.635723 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.635729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.635917 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.635950 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.636050 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.636228 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12590,"thread_start_us":12873,"threads_started":1}
I20260927 15:06:57.636559 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.615855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.0.97.2:37587)
0927 15:06:57.628107 (+ 12252us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.628119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.628748 (+ 629us) server_negotiation.cc:407] Connection header received
0927 15:06:57.628875 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.628881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.629088 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.629312 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:57.629642 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.631972 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.634129 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.635560 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.635708 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.635892 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.636111 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.636263 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12059,"thread_start_us":12270,"threads_started":1}
I20260927 15:06:57.640349 4769 tablet_service.cc:1467] Processing CreateTablet for tablet e78616a2e9ac4cf686e438ed7878de48 (DEFAULT_TABLE table=test-table [id=efebcfee9c54438b9e54b10d1bd87827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:57.641765 4769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78616a2e9ac4cf686e438ed7878de48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:57.649111 4986 tablet_bootstrap.cc:492] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Bootstrap starting.
I20260927 15:06:57.651793 4987 tablet_bootstrap.cc:492] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Bootstrap starting.
I20260927 15:06:57.652815 4986 tablet_bootstrap.cc:654] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:57.654151 4986 log.cc:824] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:57.655767 4987 tablet_bootstrap.cc:654] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:57.657290 4987 log.cc:824] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:57.658629 4989 tablet_bootstrap.cc:492] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Bootstrap starting.
I20260927 15:06:57.658806 4986 tablet_bootstrap.cc:492] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: No bootstrap required, opened a new log
I20260927 15:06:57.659118 4986 ts_tablet_manager.cc:1397] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 15:06:57.660518 4987 tablet_bootstrap.cc:492] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: No bootstrap required, opened a new log
I20260927 15:06:57.660799 4987 ts_tablet_manager.cc:1397] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:06:57.661612 4989 tablet_bootstrap.cc:654] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:57.662585 4989 log.cc:824] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:57.665675 4989 tablet_bootstrap.cc:492] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: No bootstrap required, opened a new log
I20260927 15:06:57.666036 4989 ts_tablet_manager.cc:1397] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260927 15:06:57.670240 4986 raft_consensus.cc:348] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.670749 4986 raft_consensus.cc:374] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:57.670881 4986 raft_consensus.cc:729] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad63956a0c854bf287b2938de484b2a6, State: Initialized, Role: FOLLOWER
I20260927 15:06:57.671594 4986 consensus_queue.cc:260] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.671631 4987 raft_consensus.cc:348] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.672214 4987 raft_consensus.cc:374] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:57.672410 4987 raft_consensus.cc:729] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0f0e102dea4dfc9a77ed3cfd5e7ec4, State: Initialized, Role: FOLLOWER
I20260927 15:06:57.672915 4967 heartbeater.cc:503] Master 127.0.97.62:34719 was elected leader, sending a full tablet report...
I20260927 15:06:57.673312 4987 consensus_queue.cc:260] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.674619 4989 raft_consensus.cc:348] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.675156 4989 raft_consensus.cc:374] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:57.675199 4987 ts_tablet_manager.cc:1428] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 15:06:57.675297 4989 raft_consensus.cc:729] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b06bf4ae0f4a9389003a1209fb0c42, State: Initialized, Role: FOLLOWER
I20260927 15:06:57.676107 4989 consensus_queue.cc:260] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.677204 4834 heartbeater.cc:503] Master 127.0.97.62:34719 was elected leader, sending a full tablet report...
I20260927 15:06:57.677956 4989 ts_tablet_manager.cc:1428] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 15:06:57.679703 4986 ts_tablet_manager.cc:1428] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Time spent starting tablet: real 0.020s user 0.008s sys 0.006s
W20260927 15:06:57.725732 4702 tablet.cc:2369] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:57.727035 4992 raft_consensus.cc:482] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:57.727316 4992 raft_consensus.cc:504] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.729022 4992 leader_election.cc:288] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 (127.0.97.1:43505), 66b06bf4ae0f4a9389003a1209fb0c42 (127.0.97.2:37587)
I20260927 15:06:57.735162 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.729398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43505 (local address 127.0.97.3:43205)
0927 15:06:57.729762 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.729789 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.729981 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.730427 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.730433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.730469 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:57.730662 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.730671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.731721 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.731729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.733187 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.733199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.734325 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.734331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.734509 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.734535 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.734645 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.734782 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":46}
I20260927 15:06:57.735623 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.729578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43205 (local address 127.0.97.1:43505)
0927 15:06:57.729768 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.729774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.729902 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:06:57.730187 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.730195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.730279 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.730383 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:57.730806 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.731591 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.733346 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.734126 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.734275 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.734377 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.734670 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.734799 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:06:57.735616 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.729582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42667 (local address 127.0.97.2:37587)
0927 15:06:57.729768 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.729774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.730504 (+ 730us) server_negotiation.cc:407] Connection header received
0927 15:06:57.730753 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.730758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.730815 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.730911 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:57.731454 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.732147 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.733869 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.734646 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.734815 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.734907 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.735229 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.735354 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:06:57.735713 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.729454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37587 (local address 127.0.97.3:42667)
0927 15:06:57.730393 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:57.730415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:57.730608 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:57.730923 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:57.730927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:57.730950 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:57.731289 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.731297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.732295 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.732301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.733700 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:57.733712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.734854 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.734860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:57.735034 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.735057 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:06:57.735193 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 15:06:57.735321 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":5,"thread_start_us":514,"threads_started":1}
I20260927 15:06:57.736635 4656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78616a2e9ac4cf686e438ed7878de48" candidate_uuid: "ad63956a0c854bf287b2938de484b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" is_pre_election: true
I20260927 15:06:57.736744 4789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78616a2e9ac4cf686e438ed7878de48" candidate_uuid: "ad63956a0c854bf287b2938de484b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" is_pre_election: true
I20260927 15:06:57.737206 4656 raft_consensus.cc:2454] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad63956a0c854bf287b2938de484b2a6 in term 0.
I20260927 15:06:57.737236 4789 raft_consensus.cc:2454] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad63956a0c854bf287b2938de484b2a6 in term 0.
I20260927 15:06:57.737911 4855 leader_election.cc:302] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0f0e102dea4dfc9a77ed3cfd5e7ec4, ad63956a0c854bf287b2938de484b2a6; no voters:
I20260927 15:06:57.738320 4992 raft_consensus.cc:2781] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:57.738484 4992 raft_consensus.cc:482] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:57.738587 4992 raft_consensus.cc:3037] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:57.740842 4992 raft_consensus.cc:504] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.741485 4992 leader_election.cc:288] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 (127.0.97.1:43505), 66b06bf4ae0f4a9389003a1209fb0c42 (127.0.97.2:37587)
I20260927 15:06:57.741892 4656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78616a2e9ac4cf686e438ed7878de48" candidate_uuid: "ad63956a0c854bf287b2938de484b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4"
I20260927 15:06:57.741993 4789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78616a2e9ac4cf686e438ed7878de48" candidate_uuid: "ad63956a0c854bf287b2938de484b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b06bf4ae0f4a9389003a1209fb0c42"
I20260927 15:06:57.742106 4656 raft_consensus.cc:3037] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:57.742247 4789 raft_consensus.cc:3037] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:57.744515 4789 raft_consensus.cc:2454] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad63956a0c854bf287b2938de484b2a6 in term 1.
I20260927 15:06:57.744515 4656 raft_consensus.cc:2454] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad63956a0c854bf287b2938de484b2a6 in term 1.
I20260927 15:06:57.745010 4858 leader_election.cc:302] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b06bf4ae0f4a9389003a1209fb0c42, ad63956a0c854bf287b2938de484b2a6; no voters:
I20260927 15:06:57.745333 4992 raft_consensus.cc:2781] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:57.745821 4992 raft_consensus.cc:686] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 1 LEADER]: Becoming Leader. State: Replica: ad63956a0c854bf287b2938de484b2a6, State: Running, Role: LEADER
I20260927 15:06:57.746405 4992 consensus_queue.cc:237] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } }
I20260927 15:06:57.751176 4511 catalog_manager.cc:5495] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad63956a0c854bf287b2938de484b2a6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ad63956a0c854bf287b2938de484b2a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad63956a0c854bf287b2938de484b2a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 } health_report { overall_health: UNKNOWN } } }
W20260927 15:06:57.768666 4835 tablet.cc:2369] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:57.783411 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.777537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.0.97.3:41941)
0927 15:06:57.777741 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.777747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.777893 (+ 146us) server_negotiation.cc:407] Connection header received
0927 15:06:57.778192 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.778196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.778284 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.778446 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:57.778818 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.779690 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.781131 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.781601 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.781734 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.782239 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:57.782776 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:57.782941 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.783023 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.783137 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 15:06:57.789608 4968 tablet.cc:2369] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:57.792106 4656 raft_consensus.cc:1261] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 1 FOLLOWER]: Refusing update from remote peer ad63956a0c854bf287b2938de484b2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:57.792109 4789 raft_consensus.cc:1261] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 1 FOLLOWER]: Refusing update from remote peer ad63956a0c854bf287b2938de484b2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:57.793123 4996 consensus_queue.cc:1035] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b06bf4ae0f4a9389003a1209fb0c42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:57.793488 4992 consensus_queue.cc:1035] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b0f0e102dea4dfc9a77ed3cfd5e7ec4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:57.820348 4512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:57.820636 4512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:57.824478 4512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b78170d9b748444caaaf7413d2a62715: Sending DeleteTablet for 3 replicas of tablet e78616a2e9ac4cf686e438ed7878de48
I20260927 15:06:57.825707 4902 tablet_service.cc:1514] Processing DeleteTablet for tablet e78616a2e9ac4cf686e438ed7878de48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47094
I20260927 15:06:57.825744 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet e78616a2e9ac4cf686e438ed7878de48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56102
I20260927 15:06:57.826709 5009 tablet_replica.cc:330] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: stopping tablet replica
I20260927 15:06:57.827271 5009 raft_consensus.cc:2229] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:57.827353 4769 tablet_service.cc:1514] Processing DeleteTablet for tablet e78616a2e9ac4cf686e438ed7878de48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49338
I20260927 15:06:57.828070 5009 raft_consensus.cc:2258] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:57.830569 5009 ts_tablet_manager.cc:1905] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:57.834522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4574
I20260927 15:06:57.839432 5009 ts_tablet_manager.cc:1918] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:57.839824 5009 log.cc:1196] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/e78616a2e9ac4cf686e438ed7878de48
I20260927 15:06:57.840580 5009 ts_tablet_manager.cc:1939] T e78616a2e9ac4cf686e438ed7878de48 P ad63956a0c854bf287b2938de484b2a6: Deleting consensus metadata
I20260927 15:06:57.843045 4500 catalog_manager.cc:4841] TS ad63956a0c854bf287b2938de484b2a6 (127.0.97.3:41941): tablet e78616a2e9ac4cf686e438ed7878de48 (table test-table [id=efebcfee9c54438b9e54b10d1bd87827]) successfully deleted
I20260927 15:06:57.848506 5011 tablet_replica.cc:330] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: stopping tablet replica
I20260927 15:06:57.849205 5011 raft_consensus.cc:2229] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:57.849978 5011 raft_consensus.cc:2258] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:57.850684 5010 tablet_replica.cc:330] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: stopping tablet replica
I20260927 15:06:57.851257 5010 raft_consensus.cc:2229] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:57.851847 5010 raft_consensus.cc:2258] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:57.852525 5011 ts_tablet_manager.cc:1905] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:57.854175 5010 ts_tablet_manager.cc:1905] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:06:57 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:06:57.860150 5011 ts_tablet_manager.cc:1918] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:57.860392 5011 log.cc:1196] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/e78616a2e9ac4cf686e438ed7878de48
I20260927 15:06:57.861028 5011 ts_tablet_manager.cc:1939] T e78616a2e9ac4cf686e438ed7878de48 P 66b06bf4ae0f4a9389003a1209fb0c42: Deleting consensus metadata
I20260927 15:06:57.861588 5010 ts_tablet_manager.cc:1918] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:57.863298 4500 catalog_manager.cc:4841] TS 66b06bf4ae0f4a9389003a1209fb0c42 (127.0.97.2:37587): tablet e78616a2e9ac4cf686e438ed7878de48 (table test-table [id=efebcfee9c54438b9e54b10d1bd87827]) successfully deleted
I20260927 15:06:57.863822 5010 log.cc:1196] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/e78616a2e9ac4cf686e438ed7878de48
I20260927 15:06:57.864614 5010 ts_tablet_manager.cc:1939] T e78616a2e9ac4cf686e438ed7878de48 P 4b0f0e102dea4dfc9a77ed3cfd5e7ec4: Deleting consensus metadata
I20260927 15:06:57.866992 4497 catalog_manager.cc:4841] TS 4b0f0e102dea4dfc9a77ed3cfd5e7ec4 (127.0.97.1:43505): tablet e78616a2e9ac4cf686e438ed7878de48 (table test-table [id=efebcfee9c54438b9e54b10d1bd87827]) successfully deleted
I20260927 15:06:57.868295 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:57.836953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.97.1:43505)
0927 15:06:57.837145 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:57.837152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:57.847678 (+ 10526us) server_negotiation.cc:407] Connection header received
0927 15:06:57.847919 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:57.847926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:57.848088 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:57.848207 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:57.850133 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.851340 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.853230 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:57.853901 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:57.854010 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:57.859711 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:57.859731 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:57.859754 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:57.859824 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:57.864111 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.865258 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.865268 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.865272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.865466 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:57.866029 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:57.866034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:57.866037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:57.866666 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:57.866975 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:57.867661 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 15:06:57.867891 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:06:57.869048 4696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:58.000972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4574
I20260927 15:06:58.017712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4707
Sep 27 15:06:58 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:06:58.033950 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:58.018646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.0.97.2:37587)
0927 15:06:58.018814 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:58.018822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:58.018917 (+ 95us) server_negotiation.cc:407] Connection header received
0927 15:06:58.019166 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:58.019172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:58.019259 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:58.019382 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:58.020690 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:58.021472 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:58.022721 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:58.023220 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:58.023356 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:58.027581 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:58.027594 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:58.027609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:58.027653 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:58.030742 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:58.031719 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:58.031728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:58.031730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:58.031864 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:58.032293 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:58.032298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:58.032300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:58.032710 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:58.033016 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:58.033332 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:06:58.033557 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:06:58.034782 4829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:58.125293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4707
I20260927 15:06:58.143595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4840
Sep 27 15:06:58 dist-test-slave-g0w1 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:06:58.159847 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:58.144506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.97.3:41941)
0927 15:06:58.144687 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:58.144692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:58.144717 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:58.144895 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:58.144901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:58.144990 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:58.145117 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:58.146366 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:58.147316 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:58.148589 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:58.149069 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:58.149230 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:58.153664 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:58.153675 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:58.153690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:58.153728 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:58.156902 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:58.157622 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:58.157639 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:58.157642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:58.157786 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:58.158185 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:58.158192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:58.158203 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:58.158560 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:58.158901 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:58.159196 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 15:06:58.159406 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:06:58.160825 4962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:58.283221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4840
I20260927 15:06:58.303373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 4482
I20260927 15:06:58.304623 4543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:06:58.413064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 4482
2026-09-27T15:06:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4401 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T15:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:58Z Disabled control of system clock
I20260927 15:06:58.483601 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39645
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:06:58.825582 5020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:58.825927 5020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:58.826110 5020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:58.833793 5020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:58.833922 5020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:58.834019 5020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:58.834129 5020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:58.834205 5020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:58.843900 5020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:58.845254 5020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:58.847177 5020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:58.859429 5025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:58.859468 5028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:58.859391 5026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:58.860373 5020 server_base.cc:1034] running on GCE node
I20260927 15:06:58.861322 5020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:58.862915 5020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:58.864174 5020 hybrid_clock.cc:648] HybridClock initialized: now 1790521618864094 us; error 95 us; skew 500 ppm
I20260927 15:06:58.864548 5020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:06:58.867061 5020 webserver.cc:460] Webserver started at http://127.0.97.62:46473/ using document root <none> and password file <none>
I20260927 15:06:58.867874 5020 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:58.868016 5020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:58.868379 5020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:58.871519 5020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "b386baf1858c4450abf4eb1697347105"
format_stamp: "Formatted at 2026-09-27 15:06:58 on dist-test-slave-g0w1"
server_key: "ee23607041fa704842ab6870775f10d6"
server_key_iv: "fe7291a8ddea3a7596a1461b8b180437"
server_key_version: "encryptionkey@0"
I20260927 15:06:58.872437 5020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "b386baf1858c4450abf4eb1697347105"
format_stamp: "Formatted at 2026-09-27 15:06:58 on dist-test-slave-g0w1"
server_key: "ee23607041fa704842ab6870775f10d6"
server_key_iv: "fe7291a8ddea3a7596a1461b8b180437"
server_key_version: "encryptionkey@0"
I20260927 15:06:58.878175 5020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:58.886518 5034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:58.888295 5020 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20260927 15:06:58.888463 5020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "b386baf1858c4450abf4eb1697347105"
format_stamp: "Formatted at 2026-09-27 15:06:58 on dist-test-slave-g0w1"
server_key: "ee23607041fa704842ab6870775f10d6"
server_key_iv: "fe7291a8ddea3a7596a1461b8b180437"
server_key_version: "encryptionkey@0"
I20260927 15:06:58.888672 5020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:58.894797 5020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:58.895726 5020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:58.896008 5020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:58.914227 5020 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39645
I20260927 15:06:58.914284 5085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39645 every 8 connection(s)
I20260927 15:06:58.915704 5020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:58.919708 5086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:58.924293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5020
I20260927 15:06:58.924521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:06:58.925278 388 external_mini_cluster.cc:1428] Setting key c4094a5a6bd05a626881425a5d753afc
I20260927 15:06:58.931896 5086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105: Bootstrap starting.
I20260927 15:06:58.934278 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:58.927597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.0.97.62:39645)
0927 15:06:58.928642 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:58.928654 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:58.928699 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:06:58.930453 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:58.930487 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:58.930951 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:58.931436 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:58.932600 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:58.932607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:58.932621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:58.932634 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:58.932689 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:58.932707 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:58.932939 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:58.933077 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:06:58.933338 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":373,"threads_started":1}
I20260927 15:06:58.935684 5086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:58.936779 5086 log.cc:824] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:58.939778 5086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105: No bootstrap required, opened a new log
I20260927 15:06:58.947124 5086 raft_consensus.cc:348] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b386baf1858c4450abf4eb1697347105" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39645 } }
I20260927 15:06:58.947561 5086 raft_consensus.cc:374] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:58.947666 5086 raft_consensus.cc:729] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b386baf1858c4450abf4eb1697347105, State: Initialized, Role: FOLLOWER
I20260927 15:06:58.948251 5086 consensus_queue.cc:260] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b386baf1858c4450abf4eb1697347105" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39645 } }
I20260927 15:06:58.948498 5086 raft_consensus.cc:388] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:58.948642 5086 raft_consensus.cc:482] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:58.948822 5086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:58.951367 5086 raft_consensus.cc:504] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b386baf1858c4450abf4eb1697347105" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39645 } }
I20260927 15:06:58.951875 5086 leader_election.cc:302] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b386baf1858c4450abf4eb1697347105; no voters:
I20260927 15:06:58.952500 5086 leader_election.cc:288] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:58.952642 5091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:58.953385 5091 raft_consensus.cc:686] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [term 1 LEADER]: Becoming Leader. State: Replica: b386baf1858c4450abf4eb1697347105, State: Running, Role: LEADER
I20260927 15:06:58.953976 5091 consensus_queue.cc:237] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b386baf1858c4450abf4eb1697347105" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39645 } }
I20260927 15:06:58.954571 5086 sys_catalog.cc:564] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:58.957028 5092 sys_catalog.cc:455] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b386baf1858c4450abf4eb1697347105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b386baf1858c4450abf4eb1697347105" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39645 } } }
I20260927 15:06:58.957473 5092 sys_catalog.cc:458] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:58.957321 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b386baf1858c4450abf4eb1697347105. Latest consensus state: current_term: 1 leader_uuid: "b386baf1858c4450abf4eb1697347105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b386baf1858c4450abf4eb1697347105" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39645 } } }
I20260927 15:06:58.957638 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:58.959511 5096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:58.964851 5096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:58.974632 5096 catalog_manager.cc:1347] Generated new cluster ID: 4b035078425a44b59aa668509f86213e
I20260927 15:06:58.974792 5096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:58.989820 5096 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:58.990401 5096 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:59.001988 5096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105: Generated new TSK 0
I20260927 15:06:59.003165 5096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:59.010855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39645
--builtin_ntp_servers=127.0.97.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:06:59.359192 5110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:59.359506 5110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:59.359598 5110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:59.367365 5110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:59.367482 5110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:59.367596 5110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:06:59.377292 5110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:59.378621 5110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:59.380541 5110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:59.394793 5118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:59.394816 5115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:59.395152 5116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:59.397100 5110 server_base.cc:1034] running on GCE node
I20260927 15:06:59.397903 5110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:59.399264 5110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:59.400592 5110 hybrid_clock.cc:648] HybridClock initialized: now 1790521619400541 us; error 47 us; skew 500 ppm
I20260927 15:06:59.400918 5110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:06:59.403810 5110 webserver.cc:460] Webserver started at http://127.0.97.1:39567/ using document root <none> and password file <none>
I20260927 15:06:59.404561 5110 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:59.404659 5110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:59.404932 5110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:59.408748 5110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "175ea6a6e43a4ed29e41ec663c0d88f2"
format_stamp: "Formatted at 2026-09-27 15:06:59 on dist-test-slave-g0w1"
server_key: "8680abc865aac3dd2dfe002340743b91"
server_key_iv: "43a99772ee68aec735332b93dd451f37"
server_key_version: "encryptionkey@0"
I20260927 15:06:59.409754 5110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "175ea6a6e43a4ed29e41ec663c0d88f2"
format_stamp: "Formatted at 2026-09-27 15:06:59 on dist-test-slave-g0w1"
server_key: "8680abc865aac3dd2dfe002340743b91"
server_key_iv: "43a99772ee68aec735332b93dd451f37"
server_key_version: "encryptionkey@0"
I20260927 15:06:59.416036 5110 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 15:06:59.420145 5124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:59.421798 5110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:06:59.422022 5110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "175ea6a6e43a4ed29e41ec663c0d88f2"
format_stamp: "Formatted at 2026-09-27 15:06:59 on dist-test-slave-g0w1"
server_key: "8680abc865aac3dd2dfe002340743b91"
server_key_iv: "43a99772ee68aec735332b93dd451f37"
server_key_version: "encryptionkey@0"
I20260927 15:06:59.422308 5110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:59.429139 5110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:59.430220 5110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:59.430545 5110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:59.431738 5110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:59.433568 5110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:59.433712 5110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:59.433854 5110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:59.433949 5110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:59.470046 5110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34483
I20260927 15:06:59.470186 5236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34483 every 8 connection(s)
I20260927 15:06:59.471820 5110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:59.480962 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:59.474620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56073 (local address 127.0.97.62:39645)
0927 15:06:59.475134 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:59.475142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:59.476589 (+ 1447us) server_negotiation.cc:407] Connection header received
0927 15:06:59.478412 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:59.478418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:59.478497 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:59.478623 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:59.480108 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:59.480114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:59.480119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:59.480145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:59.480187 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:59.480199 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:59.480363 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:59.480588 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:06:59.480661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260927 15:06:59.481451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5110
I20260927 15:06:59.481678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:59.482082 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:59.475365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39645 (local address 127.0.97.1:56073)
0927 15:06:59.476396 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:59.476446 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:59.478080 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:59.478892 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:59.478901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:59.479014 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:59.479769 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:59.479771 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:59.479814 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:59.479921 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:59.480312 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:59.480323 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:59.480332 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:06:59.480683 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 15:06:59.480979 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260927 15:06:59.482450 388 external_mini_cluster.cc:1428] Setting key acaa81e24f80e9f707d42a096a5e11bb
I20260927 15:06:59.483735 5237 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39645
I20260927 15:06:59.484113 5237 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:59.484799 5237 heartbeater.cc:511] Master 127.0.97.62:39645 requested a full tablet report, sending...
I20260927 15:06:59.485277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39645
--builtin_ntp_servers=127.0.97.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:06:59.487056 5051 ts_manager.cc:194] Registered new tserver with Master: 175ea6a6e43a4ed29e41ec663c0d88f2 (127.0.97.1:34483)
W20260927 15:06:59.826496 5241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:59.826805 5241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:59.826901 5241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:59.834517 5241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:59.834642 5241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:59.834756 5241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:06:59.844362 5241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:06:59.845669 5241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:59.847476 5241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:59.861819 5247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:59.861809 5246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:59.861951 5249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:59.862816 5241 server_base.cc:1034] running on GCE node
I20260927 15:06:59.863555 5241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:59.864667 5241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:59.865939 5241 hybrid_clock.cc:648] HybridClock initialized: now 1790521619865899 us; error 47 us; skew 500 ppm
I20260927 15:06:59.866300 5241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:06:59.869023 5241 webserver.cc:460] Webserver started at http://127.0.97.2:34983/ using document root <none> and password file <none>
I20260927 15:06:59.869793 5241 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:59.869937 5241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:59.870278 5241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:59.873422 5241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07"
format_stamp: "Formatted at 2026-09-27 15:06:59 on dist-test-slave-g0w1"
server_key: "960637f3cfb138372c49ec8aba052707"
server_key_iv: "6f0274dbc00a83fb225c31630dc77aa3"
server_key_version: "encryptionkey@0"
I20260927 15:06:59.874331 5241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07"
format_stamp: "Formatted at 2026-09-27 15:06:59 on dist-test-slave-g0w1"
server_key: "960637f3cfb138372c49ec8aba052707"
server_key_iv: "6f0274dbc00a83fb225c31630dc77aa3"
server_key_version: "encryptionkey@0"
I20260927 15:06:59.880445 5241 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 15:06:59.885054 5255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:59.887310 5241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:59.887566 5241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07"
format_stamp: "Formatted at 2026-09-27 15:06:59 on dist-test-slave-g0w1"
server_key: "960637f3cfb138372c49ec8aba052707"
server_key_iv: "6f0274dbc00a83fb225c31630dc77aa3"
server_key_version: "encryptionkey@0"
I20260927 15:06:59.887835 5241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:59.894057 5241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:59.894969 5241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:59.895310 5241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:59.896301 5241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:59.897768 5241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:59.897890 5241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:59.898021 5241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:59.898110 5241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:59.937516 5241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46563
I20260927 15:06:59.937614 5367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46563 every 8 connection(s)
I20260927 15:06:59.938947 5241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:59.946724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5241
I20260927 15:06:59.946938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:59.947397 388 external_mini_cluster.cc:1428] Setting key bc2c1dd9e59b121d0663c6a0902f0d2d
I20260927 15:06:59.948065 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:59.941841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42709 (local address 127.0.97.62:39645)
0927 15:06:59.942058 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:59.942063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:59.943773 (+ 1710us) server_negotiation.cc:407] Connection header received
0927 15:06:59.945559 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:59.945563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:59.945637 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:59.945740 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:59.947303 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:59.947309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:59.947314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:59.947340 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:59.947372 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:59.947383 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:59.947520 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:59.947719 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 15:06:59.947792 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:06:59.949306 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:59.942383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39645 (local address 127.0.97.2:42709)
0927 15:06:59.943585 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:59.943627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:59.945223 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:59.945964 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:59.945972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:59.946120 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:59.946886 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:59.946890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:59.946947 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:59.947108 (+ 161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:59.947481 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:59.947492 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:59.947500 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:06:59.947815 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 15:06:59.948133 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":324,"threads_started":1}
I20260927 15:06:59.950459 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39645
--builtin_ntp_servers=127.0.97.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:06:59.950877 5368 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39645
I20260927 15:06:59.951253 5368 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:59.951877 5368 heartbeater.cc:511] Master 127.0.97.62:39645 requested a full tablet report, sending...
I20260927 15:06:59.953639 5051 ts_manager.cc:194] Registered new tserver with Master: 50c7a7f8c8a44ef59819f0b2ba0e5f07 (127.0.97.2:46563)
W20260927 15:07:00.314566 5371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:00.314878 5371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:00.314970 5371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:00.322832 5371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:00.322980 5371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:00.323179 5371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:00.333132 5371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:00.334560 5371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:00.336485 5371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:00.351591 5379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:00.351665 5377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:00.351749 5376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:00.352380 5371 server_base.cc:1034] running on GCE node
I20260927 15:07:00.353106 5371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:00.354208 5371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:00.355422 5371 hybrid_clock.cc:648] HybridClock initialized: now 1790521620355390 us; error 42 us; skew 500 ppm
I20260927 15:07:00.355813 5371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:00.358512 5371 webserver.cc:460] Webserver started at http://127.0.97.3:43193/ using document root <none> and password file <none>
I20260927 15:07:00.359331 5371 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:00.359437 5371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:00.359719 5371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:00.362771 5371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ea7d63a21544e0c8eedb2e308e90c31"
format_stamp: "Formatted at 2026-09-27 15:07:00 on dist-test-slave-g0w1"
server_key: "063ef21083dda71eababce30fc80d477"
server_key_iv: "22eaf4a00254c3a0f03e72b984d8dc65"
server_key_version: "encryptionkey@0"
I20260927 15:07:00.363740 5371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ea7d63a21544e0c8eedb2e308e90c31"
format_stamp: "Formatted at 2026-09-27 15:07:00 on dist-test-slave-g0w1"
server_key: "063ef21083dda71eababce30fc80d477"
server_key_iv: "22eaf4a00254c3a0f03e72b984d8dc65"
server_key_version: "encryptionkey@0"
I20260927 15:07:00.370007 5371 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:00.374159 5385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:00.376704 5371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:00.376904 5371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "4ea7d63a21544e0c8eedb2e308e90c31"
format_stamp: "Formatted at 2026-09-27 15:07:00 on dist-test-slave-g0w1"
server_key: "063ef21083dda71eababce30fc80d477"
server_key_iv: "22eaf4a00254c3a0f03e72b984d8dc65"
server_key_version: "encryptionkey@0"
I20260927 15:07:00.377141 5371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:00.383723 5371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:00.384660 5371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:00.384961 5371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:00.385972 5371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:00.387494 5371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:00.387584 5371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:00.387669 5371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:00.387723 5371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:00.422621 5371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40953
I20260927 15:07:00.422674 5497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40953 every 8 connection(s)
I20260927 15:07:00.424085 5371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:00.426121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5371
I20260927 15:07:00.426352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:00.426900 388 external_mini_cluster.cc:1428] Setting key 2c14d83aa9f78d348181e41ad6aafe5d
I20260927 15:07:00.434394 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.427190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44265 (local address 127.0.97.62:39645)
0927 15:07:00.427410 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.427423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.429359 (+ 1936us) server_negotiation.cc:407] Connection header received
0927 15:07:00.431915 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.431921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.432008 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.432123 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.433599 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.433612 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.433616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.433634 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.433666 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.433678 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.433806 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.434028 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.434102 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:07:00.435716 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.427906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39645 (local address 127.0.97.3:44265)
0927 15:07:00.429184 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.429245 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:00.431535 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:00.432402 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:00.432411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:00.432525 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:00.433235 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:00.433238 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:00.433282 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:00.433397 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:00.433777 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:00.433790 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:00.433798 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:07:00.434124 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 15:07:00.434481 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":422,"threads_started":1}
I20260927 15:07:00.437247 5498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39645
I20260927 15:07:00.437618 5498 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:00.438302 5498 heartbeater.cc:511] Master 127.0.97.62:39645 requested a full tablet report, sending...
I20260927 15:07:00.440483 5051 ts_manager.cc:194] Registered new tserver with Master: 4ea7d63a21544e0c8eedb2e308e90c31 (127.0.97.3:40953)
I20260927 15:07:00.443463 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:07:00.451205 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.448242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.0.97.62:39645)
0927 15:07:00.448455 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.448463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.448955 (+ 492us) server_negotiation.cc:407] Connection header received
0927 15:07:00.449206 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.449218 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.449276 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.449387 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.450411 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.450419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.450428 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.450444 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.450478 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.450525 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.450669 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.450793 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.450859 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:07:00.456048 5051 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:00.463258 5051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:00.490739 5237 heartbeater.cc:503] Master 127.0.97.62:39645 was elected leader, sending a full tablet report...
I20260927 15:07:00.501312 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.477844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.0.97.1:34483)
0927 15:07:00.493693 (+ 15849us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.493707 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.494420 (+ 713us) server_negotiation.cc:407] Connection header received
0927 15:07:00.494504 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.494517 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.494868 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.495125 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.499843 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.499854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.499872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.499895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.499989 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.500020 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.500237 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.500607 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.500765 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15611,"thread_start_us":15269,"threads_started":1}
I20260927 15:07:00.502326 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.477261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52166)
0927 15:07:00.494169 (+ 16908us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.494189 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:00.494290 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:00.496931 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:00.496945 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:00.497102 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:00.499362 (+ 2260us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:00.499369 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:00.499423 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:00.499560 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:00.500309 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:00.500328 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:00.500357 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:07:00.501701 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0927 15:07:00.501848 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16767,"thread_start_us":16607,"threads_started":1}
I20260927 15:07:00.504874 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.477147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40953 (local address 127.0.0.1:54174)
0927 15:07:00.493266 (+ 16119us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.493343 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:00.493542 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:00.499184 (+ 5642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:00.499198 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:00.499239 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:00.500265 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:00.500271 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:00.500295 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:00.500373 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:00.504189 (+ 3816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:00.504195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:00.504204 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:07:00.504353 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 15:07:00.504411 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15738,"thread_start_us":14786,"threads_started":1}
I20260927 15:07:00.504858 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.477265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.0.97.3:40953)
0927 15:07:00.498312 (+ 21047us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.498326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.498436 (+ 110us) server_negotiation.cc:407] Connection header received
0927 15:07:00.498531 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.498544 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.498876 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.502592 (+ 3716us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.503863 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.503875 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.503892 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.503916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.503979 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.504005 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.504275 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.504344 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.504447 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20772,"thread_start_us":20812,"threads_started":1}
I20260927 15:07:00.506012 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.494745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.97.2:46563)
0927 15:07:00.502271 (+ 7526us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.502280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.502316 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:07:00.502394 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.502404 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.502692 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.502863 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.505047 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.505057 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.505072 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.505109 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.505170 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.505192 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.505318 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.505586 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.505686 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7342,"thread_start_us":8608,"threads_started":1}
I20260927 15:07:00.506036 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.494319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46563 (local address 127.0.0.1:43346)
0927 15:07:00.502045 (+ 7726us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.502071 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:00.502197 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:00.503876 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:00.503889 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:00.503917 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:00.504751 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:00.504754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:00.504776 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:00.504827 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:00.505395 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:00.505402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:00.505407 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:07:00.505544 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 15:07:00.505595 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7570,"thread_start_us":7979,"threads_started":1}
I20260927 15:07:00.507395 5172 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa423511de7469e9da0901e4646c1f3 (DEFAULT_TABLE table=test-table [id=ca14a01966cb4057ae8661f9ad6fd0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:00.508702 5172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa423511de7469e9da0901e4646c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:00.509019 5433 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa423511de7469e9da0901e4646c1f3 (DEFAULT_TABLE table=test-table [id=ca14a01966cb4057ae8661f9ad6fd0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:00.509968 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa423511de7469e9da0901e4646c1f3 (DEFAULT_TABLE table=test-table [id=ca14a01966cb4057ae8661f9ad6fd0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:00.510639 5433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa423511de7469e9da0901e4646c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:00.511385 5303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa423511de7469e9da0901e4646c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:00.521453 5516 tablet_bootstrap.cc:492] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: Bootstrap starting.
I20260927 15:07:00.525045 5516 tablet_bootstrap.cc:654] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:00.526427 5516 log.cc:824] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:00.527032 5518 tablet_bootstrap.cc:492] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Bootstrap starting.
I20260927 15:07:00.527859 5517 tablet_bootstrap.cc:492] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Bootstrap starting.
I20260927 15:07:00.530110 5516 tablet_bootstrap.cc:492] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: No bootstrap required, opened a new log
I20260927 15:07:00.530441 5516 ts_tablet_manager.cc:1397] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 15:07:00.531482 5517 tablet_bootstrap.cc:654] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:00.532656 5517 log.cc:824] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:00.532655 5518 tablet_bootstrap.cc:654] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:00.534090 5518 log.cc:824] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:00.536013 5517 tablet_bootstrap.cc:492] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: No bootstrap required, opened a new log
I20260927 15:07:00.536307 5517 ts_tablet_manager.cc:1397] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260927 15:07:00.537559 5518 tablet_bootstrap.cc:492] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: No bootstrap required, opened a new log
I20260927 15:07:00.537870 5518 ts_tablet_manager.cc:1397] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 15:07:00.541530 5516 raft_consensus.cc:348] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.542184 5516 raft_consensus.cc:374] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:00.542325 5516 raft_consensus.cc:729] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175ea6a6e43a4ed29e41ec663c0d88f2, State: Initialized, Role: FOLLOWER
I20260927 15:07:00.543123 5516 consensus_queue.cc:260] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.544325 5517 raft_consensus.cc:348] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.544961 5517 raft_consensus.cc:374] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:00.545208 5517 raft_consensus.cc:729] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c7a7f8c8a44ef59819f0b2ba0e5f07, State: Initialized, Role: FOLLOWER
I20260927 15:07:00.546176 5516 ts_tablet_manager.cc:1428] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 15:07:00.546092 5517 consensus_queue.cc:260] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.547374 5368 heartbeater.cc:503] Master 127.0.97.62:39645 was elected leader, sending a full tablet report...
I20260927 15:07:00.548161 5517 ts_tablet_manager.cc:1428] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 15:07:00.551353 5518 raft_consensus.cc:348] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.551913 5518 raft_consensus.cc:374] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:00.552029 5518 raft_consensus.cc:729] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea7d63a21544e0c8eedb2e308e90c31, State: Initialized, Role: FOLLOWER
I20260927 15:07:00.552703 5518 consensus_queue.cc:260] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.553810 5498 heartbeater.cc:503] Master 127.0.97.62:39645 was elected leader, sending a full tablet report...
I20260927 15:07:00.554271 5518 ts_tablet_manager.cc:1428] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 15:07:00.661252 5524 raft_consensus.cc:482] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:00.661628 5524 raft_consensus.cc:504] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.663405 5524 leader_election.cc:288] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 175ea6a6e43a4ed29e41ec663c0d88f2 (127.0.97.1:34483), 50c7a7f8c8a44ef59819f0b2ba0e5f07 (127.0.97.2:46563)
I20260927 15:07:00.666291 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.663866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37757 (local address 127.0.97.1:34483)
0927 15:07:00.664070 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.664075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.664209 (+ 134us) server_negotiation.cc:407] Connection header received
0927 15:07:00.664400 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.664408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.664497 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.664626 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.665634 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.665640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.665645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.665664 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.665703 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.665715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.665846 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.665972 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.666034 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:07:00.666290 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.663912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.97.3:37757)
0927 15:07:00.664103 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.664128 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:00.664256 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:00.664655 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:00.664662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:00.664689 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:00.665420 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:00.665422 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:00.665437 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:00.665479 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:00.665810 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:00.665814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:00.665825 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:07:00.665986 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:07:00.666041 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:07:00.667559 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.664661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51251 (local address 127.0.97.2:46563)
0927 15:07:00.664871 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.664878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.665342 (+ 464us) server_negotiation.cc:407] Connection header received
0927 15:07:00.665541 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.665548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.665604 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.665711 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.666970 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.666974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.666978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.666993 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.667022 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.667034 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.667204 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.667292 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.667337 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:07:00.667366 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa423511de7469e9da0901e4646c1f3" candidate_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" is_pre_election: true
I20260927 15:07:00.667855 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.664358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46563 (local address 127.0.97.3:51251)
0927 15:07:00.665235 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.665249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:00.665400 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:00.665736 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:00.665740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:00.665763 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:00.666754 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:00.666757 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:00.666774 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:00.666822 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:00.667178 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:00.667184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:00.667189 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:07:00.667310 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 15:07:00.667373 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":543,"threads_started":1}
I20260927 15:07:00.668015 5192 raft_consensus.cc:2454] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea7d63a21544e0c8eedb2e308e90c31 in term 0.
I20260927 15:07:00.668761 5387 leader_election.cc:302] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 175ea6a6e43a4ed29e41ec663c0d88f2, 4ea7d63a21544e0c8eedb2e308e90c31; no voters:
I20260927 15:07:00.669260 5524 raft_consensus.cc:2781] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:00.669422 5524 raft_consensus.cc:482] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:00.669541 5524 raft_consensus.cc:3037] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:00.669399 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa423511de7469e9da0901e4646c1f3" candidate_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" is_pre_election: true
I20260927 15:07:00.669953 5323 raft_consensus.cc:2454] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea7d63a21544e0c8eedb2e308e90c31 in term 0.
I20260927 15:07:00.672474 5524 raft_consensus.cc:504] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.673166 5524 leader_election.cc:288] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [CANDIDATE]: Term 1 election: Requested vote from peers 175ea6a6e43a4ed29e41ec663c0d88f2 (127.0.97.1:34483), 50c7a7f8c8a44ef59819f0b2ba0e5f07 (127.0.97.2:46563)
I20260927 15:07:00.673767 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa423511de7469e9da0901e4646c1f3" candidate_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2"
I20260927 15:07:00.673822 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa423511de7469e9da0901e4646c1f3" candidate_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07"
I20260927 15:07:00.674016 5323 raft_consensus.cc:3037] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:00.674017 5192 raft_consensus.cc:3037] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 15:07:00.675866 5499 tablet.cc:2369] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:00.677448 5323 raft_consensus.cc:2454] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea7d63a21544e0c8eedb2e308e90c31 in term 1.
I20260927 15:07:00.677448 5192 raft_consensus.cc:2454] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea7d63a21544e0c8eedb2e308e90c31 in term 1.
I20260927 15:07:00.677917 5387 leader_election.cc:302] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea7d63a21544e0c8eedb2e308e90c31, 50c7a7f8c8a44ef59819f0b2ba0e5f07; no voters:
I20260927 15:07:00.678236 5524 raft_consensus.cc:2781] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:00.678735 5524 raft_consensus.cc:686] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea7d63a21544e0c8eedb2e308e90c31, State: Running, Role: LEADER
I20260927 15:07:00.679402 5524 consensus_queue.cc:237] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } }
I20260927 15:07:00.684271 5051 catalog_manager.cc:5495] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ea7d63a21544e0c8eedb2e308e90c31 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea7d63a21544e0c8eedb2e308e90c31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 } health_report { overall_health: UNKNOWN } } }
W20260927 15:07:00.691336 5369 tablet.cc:2369] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:00.724300 5238 tablet.cc:2369] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:00.729321 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.726688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.3:40953)
0927 15:07:00.726982 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.726990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.727045 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:07:00.727338 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.727342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.727428 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.727542 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.728676 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.728681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.728684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.728701 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.728743 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.728756 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.728919 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.729008 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.729071 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:07:00.737943 5323 raft_consensus.cc:1261] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 1 FOLLOWER]: Refusing update from remote peer 4ea7d63a21544e0c8eedb2e308e90c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:00.738008 5192 raft_consensus.cc:1261] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 1 FOLLOWER]: Refusing update from remote peer 4ea7d63a21544e0c8eedb2e308e90c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:00.739176 5526 consensus_queue.cc:1035] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c7a7f8c8a44ef59819f0b2ba0e5f07" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:00.739624 5524 consensus_queue.cc:1035] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "175ea6a6e43a4ed29e41ec663c0d88f2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:00.761813 5529 mvcc.cc:205] Tried to move back new op lower bound from 7333976558532517888 to 7333976558307287040. Current Snapshot: MvccSnapshot[applied={T|T < 7333976558532517888 or (T in {7333976558532517888})}]
I20260927 15:07:00.771839 5051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:00.772156 5051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:00.776209 5051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b386baf1858c4450abf4eb1697347105: Sending DeleteTablet for 3 replicas of tablet 9fa423511de7469e9da0901e4646c1f3
I20260927 15:07:00.777565 5433 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa423511de7469e9da0901e4646c1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20260927 15:07:00.777647 5172 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa423511de7469e9da0901e4646c1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:00 UTC) from {username='slave'} at 127.0.0.1:52166
I20260927 15:07:00.777702 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa423511de7469e9da0901e4646c1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:00 UTC) from {username='slave'} at 127.0.0.1:43346
I20260927 15:07:00.780339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5110
I20260927 15:07:00.791464 5540 tablet_replica.cc:330] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: stopping tablet replica
I20260927 15:07:00.792335 5540 raft_consensus.cc:2229] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:00.793424 5540 raft_consensus.cc:2258] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:00.795694 5539 tablet_replica.cc:330] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2: stopping tablet replica
I20260927 15:07:00.796442 5541 tablet_replica.cc:330] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: stopping tablet replica
I20260927 15:07:00.797051 5541 raft_consensus.cc:2229] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:00.797734 5541 raft_consensus.cc:2258] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:00.799418 5540 ts_tablet_manager.cc:1905] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:00.800109 5539 raft_consensus.cc:2229] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:00.800397 5541 ts_tablet_manager.cc:1905] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:00.801237 5539 raft_consensus.cc:2258] T 9fa423511de7469e9da0901e4646c1f3 P 175ea6a6e43a4ed29e41ec663c0d88f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:00.802299 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.782280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.97.1:34483)
0927 15:07:00.782497 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.782507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.798161 (+ 15654us) server_negotiation.cc:407] Connection header received
0927 15:07:00.798691 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.798700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.798814 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.798894 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.801474 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.801481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.801486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.801511 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.801553 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.801570 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.801673 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.801918 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.801997 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:07:00.803526 5232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:00.806924 5540 ts_tablet_manager.cc:1918] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:00.807260 5540 log.cc:1196] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/9fa423511de7469e9da0901e4646c1f3
I20260927 15:07:00.807914 5540 ts_tablet_manager.cc:1939] T 9fa423511de7469e9da0901e4646c1f3 P 4ea7d63a21544e0c8eedb2e308e90c31: Deleting consensus metadata
I20260927 15:07:00.809948 5035 catalog_manager.cc:4841] TS 4ea7d63a21544e0c8eedb2e308e90c31 (127.0.97.3:40953): tablet 9fa423511de7469e9da0901e4646c1f3 (table test-table [id=ca14a01966cb4057ae8661f9ad6fd0d4]) successfully deleted
I20260927 15:07:00.810523 5541 ts_tablet_manager.cc:1918] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:00.810745 5541 log.cc:1196] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/9fa423511de7469e9da0901e4646c1f3
I20260927 15:07:00.811368 5541 ts_tablet_manager.cc:1939] T 9fa423511de7469e9da0901e4646c1f3 P 50c7a7f8c8a44ef59819f0b2ba0e5f07: Deleting consensus metadata
I20260927 15:07:00.813611 5036 catalog_manager.cc:4841] TS 50c7a7f8c8a44ef59819f0b2ba0e5f07 (127.0.97.2:46563): tablet 9fa423511de7469e9da0901e4646c1f3 (table test-table [id=ca14a01966cb4057ae8661f9ad6fd0d4]) successfully deleted
I20260927 15:07:00.902361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5110
I20260927 15:07:00.925678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5241
W20260927 15:07:00.925751 5036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34483 (error 108)
I20260927 15:07:00.930397 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.926895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.97.2:46563)
0927 15:07:00.927109 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:00.927116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:00.927566 (+ 450us) server_negotiation.cc:407] Connection header received
0927 15:07:00.927641 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:00.927647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:00.927759 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:00.927853 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:00.929120 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:00.929127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:00.929132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:00.929155 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:00.929195 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:00.929211 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:00.929342 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:00.929898 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 15:07:00.930078 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:00.931567 5363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:00.942775 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.942249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52180)
0927 15:07:00.942498 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.942626 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 15:07:00.943215 5036 catalog_manager.cc:4584] TS 175ea6a6e43a4ed29e41ec663c0d88f2 (127.0.97.1:34483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fa423511de7469e9da0901e4646c1f3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
I20260927 15:07:00.997218 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.996631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52190)
0927 15:07:00.996889 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.997008 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 15:07:00.998495 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:00.998061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52206)
0927 15:07:00.998240 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:00.998334 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:07:01.044836 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5241
I20260927 15:07:01.059160 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.058579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52208)
0927 15:07:01.058823 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:01.058948 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 15:07:01.060606 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.060016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52210)
0927 15:07:01.060249 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:01.060385 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 15:07:01.063936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5371
I20260927 15:07:01.067449 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.064806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.0.97.3:40953)
0927 15:07:01.065001 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:01.065007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:01.065026 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:07:01.065210 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:01.065214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:01.065268 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:01.065367 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:01.066460 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:01.066465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:01.066468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:01.066486 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:01.066520 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:01.066541 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:01.066670 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:01.066881 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:07:01.066983 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:07:01.068248 5493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:01.141295 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.140804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52214)
0927 15:07:01.141035 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:01.141128 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 15:07:01.142513 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.142093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52226)
0927 15:07:01.142297 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:01.142378 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 15:07:01.189141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5371
I20260927 15:07:01.208343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5020
I20260927 15:07:01.209614 5081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:01.317291 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.316625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34483 (local address 127.0.0.1:52230)
0927 15:07:01.316967 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:01.317061 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 15:07:01.317250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5020
2026-09-27T15:07:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2902 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 15:07:01.359737 388 mini_oidc.cc:152] Starting JWKS server
I20260927 15:07:01.362345 388 webserver.cc:460] Webserver started at https://127.0.0.1:42425/ using document root <none> and password file <none>
I20260927 15:07:01.362660 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:07:01.363370 388 webserver.cc:460] Webserver started at http://127.0.0.1:36469/ using document root <none> and password file <none>
2026-09-27T15:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:01Z Disabled control of system clock
I20260927 15:07:01.392058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46371
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46371
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/testchainca.cert with env {}
W20260927 15:07:01.751415 5553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:01.751734 5553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:01.751822 5553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:01.759775 5553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:01.759907 5553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:01.759948 5553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:01.759997 5553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:01.760048 5553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:01.760113 5553 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:01.760154 5553 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42425/jwks.json
I20260927 15:07:01.770447 5553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:01.771785 5553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:01.773247 5553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:01.786768 5559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:01.786782 5561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:01.786770 5558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:01.787510 5553 server_base.cc:1034] running on GCE node
I20260927 15:07:01.788434 5553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:01.790078 5553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:01.791332 5553 hybrid_clock.cc:648] HybridClock initialized: now 1790521621791280 us; error 51 us; skew 500 ppm
I20260927 15:07:01.791751 5553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:01.794288 5553 webserver.cc:460] Webserver started at http://127.0.97.62:42127/ using document root <none> and password file <none>
I20260927 15:07:01.795152 5553 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:01.795301 5553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:01.795640 5553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:01.798789 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "59e8d84c0bc143579aa5a10119f5f576"
format_stamp: "Formatted at 2026-09-27 15:07:01 on dist-test-slave-g0w1"
server_key: "320233535595c7df0a4e885dc7be4a3b"
server_key_iv: "69d8d3af731c89c461a030c016a1cf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:01.799788 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "59e8d84c0bc143579aa5a10119f5f576"
format_stamp: "Formatted at 2026-09-27 15:07:01 on dist-test-slave-g0w1"
server_key: "320233535595c7df0a4e885dc7be4a3b"
server_key_iv: "69d8d3af731c89c461a030c016a1cf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:01.805660 5553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:01.809861 5567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:01.811654 5553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:07:01.811887 5553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "59e8d84c0bc143579aa5a10119f5f576"
format_stamp: "Formatted at 2026-09-27 15:07:01 on dist-test-slave-g0w1"
server_key: "320233535595c7df0a4e885dc7be4a3b"
server_key_iv: "69d8d3af731c89c461a030c016a1cf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:01.812140 5553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:01.824652 5553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:01.825570 5553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:01.825887 5553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:01.846465 5553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46371
I20260927 15:07:01.846488 5618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46371 every 8 connection(s)
I20260927 15:07:01.848203 5553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:01.852298 5619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:01.853575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5553
I20260927 15:07:01.853778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:01.854595 388 external_mini_cluster.cc:1428] Setting key 182819797fbfedf52064a277ed946011
I20260927 15:07:01.866350 5619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576: Bootstrap starting.
I20260927 15:07:01.868999 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.856745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.0.97.62:46371)
0927 15:07:01.857861 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:01.857873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:01.857918 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:07:01.859634 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:01.859659 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:01.860220 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:01.860675 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:01.861280 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:01.862590 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:01.864152 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:01.864897 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:01.865123 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:01.866401 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:01.866409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:01.866425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:01.867137 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:01.867232 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:01.867256 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:01.867386 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:01.867627 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:07:01.867964 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":331,"threads_started":1}
I20260927 15:07:01.870378 5619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:01.871726 5619 log.cc:824] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:01.873750 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:01.875274 5619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576: No bootstrap required, opened a new log
I20260927 15:07:01.882758 5619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8d84c0bc143579aa5a10119f5f576" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46371 } }
I20260927 15:07:01.883252 5619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:01.883373 5619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e8d84c0bc143579aa5a10119f5f576, State: Initialized, Role: FOLLOWER
I20260927 15:07:01.883996 5619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8d84c0bc143579aa5a10119f5f576" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46371 } }
I20260927 15:07:01.884258 5619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:01.884410 5619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:01.884598 5619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:01.886853 5619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8d84c0bc143579aa5a10119f5f576" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46371 } }
I20260927 15:07:01.887562 5619 leader_election.cc:302] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e8d84c0bc143579aa5a10119f5f576; no voters:
I20260927 15:07:01.888209 5619 leader_election.cc:288] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:01.888391 5624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:01.889237 5624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [term 1 LEADER]: Becoming Leader. State: Replica: 59e8d84c0bc143579aa5a10119f5f576, State: Running, Role: LEADER
I20260927 15:07:01.889891 5624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8d84c0bc143579aa5a10119f5f576" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46371 } }
I20260927 15:07:01.890586 5619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:01.893090 5625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e8d84c0bc143579aa5a10119f5f576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8d84c0bc143579aa5a10119f5f576" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46371 } } }
I20260927 15:07:01.893509 5625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:01.894974 5626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e8d84c0bc143579aa5a10119f5f576. Latest consensus state: current_term: 1 leader_uuid: "59e8d84c0bc143579aa5a10119f5f576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8d84c0bc143579aa5a10119f5f576" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46371 } } }
I20260927 15:07:01.895396 5626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:01.897938 5629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:01.904603 5629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:01.915427 5629 catalog_manager.cc:1347] Generated new cluster ID: 3162e7670cec410d8855b4bb6b9147b9
I20260927 15:07:01.915670 5629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:01.930505 5629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:01.932042 5629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:01.939505 5629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e8d84c0bc143579aa5a10119f5f576: Generated new TSK 0
I20260927 15:07:01.940505 5629 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:07:01.994304 5649 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 15:07:01.994504 5649 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:07:01.995410 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:01.964933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.0.97.62:46371)
0927 15:07:01.965176 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:01.965186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:01.965950 (+ 764us) server_negotiation.cc:407] Connection header received
0927 15:07:01.966362 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:01.966369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:01.966472 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:01.966649 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:01.967489 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:01.968339 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:01.970630 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:01.971248 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:01.971371 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:01.972157 (+ 786us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:01.994760 (+ 22603us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:07:01.995170 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 15:07:01.995571 5622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 15:07:01.997318 5648 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:01.964785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46371 (local address 127.0.0.1:44934)
0927 15:07:01.965819 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:01.965835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:01.966075 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:01.966724 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:01.966730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:01.966746 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:01.967239 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:01.967255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:01.968516 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:01.968523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:01.970369 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:01.970389 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:01.971394 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:01.971469 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:01.971676 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:01.971777 (+ 101us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:01.995064 (+ 23287us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:07:01.995459 (+ 395us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46371: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":667,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:46371: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 15:07:01.998857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5553
I20260927 15:07:02.000280 5614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:02.094835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5553
2026-09-27T15:07:02Z chronyd exiting
I20260927 15:07:02.146441 388 test_util.cc:175] -----------------------------------------------
I20260927 15:07:02.146574 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521577514430-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (791 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 15:07:02.150454 388 mini_oidc.cc:152] Starting JWKS server
I20260927 15:07:02.152314 388 webserver.cc:460] Webserver started at https://127.0.0.1:34957/ using document root <none> and password file <none>
I20260927 15:07:02.152556 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:07:02.153139 388 webserver.cc:460] Webserver started at http://127.0.0.1:33513/ using document root <none> and password file <none>
2026-09-27T15:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:02Z Disabled control of system clock
I20260927 15:07:02.181250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40441
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40441
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34957/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/testchainca.cert with env {}
W20260927 15:07:02.526357 5656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:02.526693 5656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:02.526782 5656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:02.534425 5656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:02.534552 5656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:02.534590 5656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:02.534642 5656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:02.534688 5656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:02.534744 5656 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:02.534783 5656 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34957/jwks.json
I20260927 15:07:02.545342 5656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34957/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:02.546648 5656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:02.547994 5656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:02.560930 5661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:02.560999 5662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:02.560959 5664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:02.562407 5656 server_base.cc:1034] running on GCE node
I20260927 15:07:02.563755 5656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:02.565383 5656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:02.566670 5656 hybrid_clock.cc:648] HybridClock initialized: now 1790521622566621 us; error 54 us; skew 500 ppm
I20260927 15:07:02.567124 5656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:02.573871 5656 webserver.cc:460] Webserver started at http://127.0.97.62:38293/ using document root <none> and password file <none>
I20260927 15:07:02.574816 5656 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:02.574980 5656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:02.575389 5656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:02.578491 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "6e594ddc05ea4032a695759d9dd511c2"
format_stamp: "Formatted at 2026-09-27 15:07:02 on dist-test-slave-g0w1"
server_key: "dc489645492a268cb36ddfc0f948ba6e"
server_key_iv: "ca8485a56b34f9438c0a5f97ed43229b"
server_key_version: "encryptionkey@0"
I20260927 15:07:02.579504 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e594ddc05ea4032a695759d9dd511c2"
format_stamp: "Formatted at 2026-09-27 15:07:02 on dist-test-slave-g0w1"
server_key: "dc489645492a268cb36ddfc0f948ba6e"
server_key_iv: "ca8485a56b34f9438c0a5f97ed43229b"
server_key_version: "encryptionkey@0"
I20260927 15:07:02.585593 5656 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:02.589761 5670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:02.591566 5656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:07:02.591799 5656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "6e594ddc05ea4032a695759d9dd511c2"
format_stamp: "Formatted at 2026-09-27 15:07:02 on dist-test-slave-g0w1"
server_key: "dc489645492a268cb36ddfc0f948ba6e"
server_key_iv: "ca8485a56b34f9438c0a5f97ed43229b"
server_key_version: "encryptionkey@0"
I20260927 15:07:02.592156 5656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:02.607053 5656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:02.607997 5656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:02.608300 5656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:02.629133 5656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40441
I20260927 15:07:02.629184 5721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40441 every 8 connection(s)
I20260927 15:07:02.630873 5656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:02.633294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5656
I20260927 15:07:02.633523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:02.634359 388 external_mini_cluster.cc:1428] Setting key f662bc6f63000ca69947f5ead3629044
I20260927 15:07:02.636355 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:02.649139 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:02.636510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.0.97.62:40441)
0927 15:07:02.637645 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:02.637666 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:02.637713 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:07:02.639445 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:02.639470 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:02.640032 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:02.640532 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:02.641176 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:02.642609 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:02.644105 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:02.644852 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:02.645082 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:02.646367 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:02.646375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:02.646393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:02.647046 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:02.647227 (+ 181us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:02.647255 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:02.647414 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:02.647662 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:07:02.648021 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":425,"threads_started":1}
I20260927 15:07:02.650223 5722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2: Bootstrap starting.
I20260927 15:07:02.653653 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:02.654062 5722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:02.655212 5722 log.cc:824] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:02.658499 5722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2: No bootstrap required, opened a new log
I20260927 15:07:02.666311 5722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e594ddc05ea4032a695759d9dd511c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40441 } }
I20260927 15:07:02.666733 5722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:02.666874 5722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e594ddc05ea4032a695759d9dd511c2, State: Initialized, Role: FOLLOWER
I20260927 15:07:02.667565 5722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e594ddc05ea4032a695759d9dd511c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40441 } }
I20260927 15:07:02.667806 5722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:02.667963 5722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:02.668159 5722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:02.670452 5722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e594ddc05ea4032a695759d9dd511c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40441 } }
I20260927 15:07:02.671166 5722 leader_election.cc:302] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e594ddc05ea4032a695759d9dd511c2; no voters:
I20260927 15:07:02.671824 5722 leader_election.cc:288] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:02.672061 5727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:02.673044 5727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [term 1 LEADER]: Becoming Leader. State: Replica: 6e594ddc05ea4032a695759d9dd511c2, State: Running, Role: LEADER
I20260927 15:07:02.673573 5727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e594ddc05ea4032a695759d9dd511c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40441 } }
I20260927 15:07:02.673868 5722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:02.676601 5729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e594ddc05ea4032a695759d9dd511c2. Latest consensus state: current_term: 1 leader_uuid: "6e594ddc05ea4032a695759d9dd511c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e594ddc05ea4032a695759d9dd511c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40441 } } }
I20260927 15:07:02.677156 5729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:02.676816 5728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e594ddc05ea4032a695759d9dd511c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e594ddc05ea4032a695759d9dd511c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40441 } } }
I20260927 15:07:02.677580 5728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:02.681818 5736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:02.687525 5736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:02.697376 5736 catalog_manager.cc:1347] Generated new cluster ID: 4247874a1104463db02f50ffa68cee1a
I20260927 15:07:02.697608 5736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:02.723368 5736 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:02.724735 5736 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:02.736501 5736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e594ddc05ea4032a695759d9dd511c2: Generated new TSK 0
I20260927 15:07:02.737406 5736 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:07:02.777254 5752 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 15:07:02.777400 5752 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:07:02.778746 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:02.757470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.97.62:40441)
0927 15:07:02.757666 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:02.757673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:02.758149 (+ 476us) server_negotiation.cc:407] Connection header received
0927 15:07:02.758353 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:02.758360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:02.758484 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:02.758668 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:02.759204 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:02.760328 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:02.761843 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:02.762562 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:02.762698 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:02.763312 (+ 614us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:02.777989 (+ 14677us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:07:02.778414 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:07:02.778780 5751 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:02.757289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40441 (local address 127.0.0.1:41600)
0927 15:07:02.758042 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:02.758058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:02.758186 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:02.758681 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:02.758685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:02.758694 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:02.758935 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:02.758942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:02.760554 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:02.760560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:02.761680 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:02.761694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:02.762728 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:02.762774 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:02.762942 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:02.762981 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:02.778257 (+ 15276us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:07:02.778467 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":455,"threads_started":1}
W20260927 15:07:02.778878 5725 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 15:07:02.780048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5656
I20260927 15:07:02.781563 5717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:02.876124 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5656
2026-09-27T15:07:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (783 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 15:07:02.934343 388 mini_oidc.cc:152] Starting JWKS server
I20260927 15:07:02.936376 388 webserver.cc:460] Webserver started at https://127.0.0.1:44795/ using document root <none> and password file <none>
I20260927 15:07:02.936589 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:07:02.937235 388 webserver.cc:460] Webserver started at http://127.0.0.1:43187/ using document root <none> and password file <none>
2026-09-27T15:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:02Z Disabled control of system clock
I20260927 15:07:02.965396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33245
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33245
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44795/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:07:03.305465 5759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:03.305785 5759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:03.305872 5759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:03.314072 5759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:03.314227 5759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:03.314268 5759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:03.314320 5759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:03.314366 5759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:03.314424 5759 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:03.314465 5759 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44795/jwks.json
I20260927 15:07:03.324620 5759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44795/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:03.326063 5759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:03.328318 5759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:03.341408 5764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:03.341552 5765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:03.341854 5767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:03.342396 5759 server_base.cc:1034] running on GCE node
I20260927 15:07:03.343434 5759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:03.345139 5759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:03.346457 5759 hybrid_clock.cc:648] HybridClock initialized: now 1790521623346406 us; error 51 us; skew 500 ppm
I20260927 15:07:03.347229 5759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:03.350117 5759 webserver.cc:460] Webserver started at http://127.0.97.62:44963/ using document root <none> and password file <none>
I20260927 15:07:03.350903 5759 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:03.351061 5759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:03.351472 5759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:03.354750 5759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "472253de659040c2bdaf4abbc8a6b187"
format_stamp: "Formatted at 2026-09-27 15:07:03 on dist-test-slave-g0w1"
server_key: "1bc0bd03d165f6f5c876a6d8e4818500"
server_key_iv: "b37bf663fdda7b77807c191004c26456"
server_key_version: "encryptionkey@0"
I20260927 15:07:03.355860 5759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "472253de659040c2bdaf4abbc8a6b187"
format_stamp: "Formatted at 2026-09-27 15:07:03 on dist-test-slave-g0w1"
server_key: "1bc0bd03d165f6f5c876a6d8e4818500"
server_key_iv: "b37bf663fdda7b77807c191004c26456"
server_key_version: "encryptionkey@0"
I20260927 15:07:03.361594 5759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:03.365624 5773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:03.367453 5759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:03.367669 5759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "472253de659040c2bdaf4abbc8a6b187"
format_stamp: "Formatted at 2026-09-27 15:07:03 on dist-test-slave-g0w1"
server_key: "1bc0bd03d165f6f5c876a6d8e4818500"
server_key_iv: "b37bf663fdda7b77807c191004c26456"
server_key_version: "encryptionkey@0"
I20260927 15:07:03.367925 5759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:03.392882 5759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:03.393792 5759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:03.394100 5759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:03.413435 5759 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33245
I20260927 15:07:03.413453 5824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33245 every 8 connection(s)
I20260927 15:07:03.414907 5759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:03.417369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5759
I20260927 15:07:03.417609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:03.418424 388 external_mini_cluster.cc:1428] Setting key 31ea9729fb4fdcdfe25c8cf2ceabaf2a
I20260927 15:07:03.419827 5825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:03.433370 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:03.420640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.0.97.62:33245)
0927 15:07:03.421706 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:03.421718 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:03.421761 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:07:03.423438 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:03.423464 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:03.424037 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:03.424503 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:03.425100 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:03.426486 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:03.427954 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:03.428677 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:03.428910 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:03.430266 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:03.430275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:03.430292 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:03.430963 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:03.431053 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:03.431132 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:03.431272 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:03.431534 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 15:07:03.431891 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":382,"threads_started":1}
I20260927 15:07:03.433480 5825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187: Bootstrap starting.
I20260927 15:07:03.437387 5825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:03.437849 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:03.438887 5825 log.cc:824] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:03.442221 5825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187: No bootstrap required, opened a new log
I20260927 15:07:03.449919 5825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472253de659040c2bdaf4abbc8a6b187" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33245 } }
I20260927 15:07:03.450419 5825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:03.450534 5825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472253de659040c2bdaf4abbc8a6b187, State: Initialized, Role: FOLLOWER
I20260927 15:07:03.451254 5825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472253de659040c2bdaf4abbc8a6b187" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33245 } }
I20260927 15:07:03.451488 5825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:03.451627 5825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:03.451795 5825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:03.453990 5825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472253de659040c2bdaf4abbc8a6b187" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33245 } }
I20260927 15:07:03.454566 5825 leader_election.cc:302] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472253de659040c2bdaf4abbc8a6b187; no voters:
I20260927 15:07:03.455346 5825 leader_election.cc:288] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:03.455546 5830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:03.456384 5830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [term 1 LEADER]: Becoming Leader. State: Replica: 472253de659040c2bdaf4abbc8a6b187, State: Running, Role: LEADER
I20260927 15:07:03.457026 5830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472253de659040c2bdaf4abbc8a6b187" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33245 } }
I20260927 15:07:03.457669 5825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:03.459736 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472253de659040c2bdaf4abbc8a6b187. Latest consensus state: current_term: 1 leader_uuid: "472253de659040c2bdaf4abbc8a6b187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472253de659040c2bdaf4abbc8a6b187" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33245 } } }
I20260927 15:07:03.460134 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:03.460711 5831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472253de659040c2bdaf4abbc8a6b187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472253de659040c2bdaf4abbc8a6b187" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33245 } } }
I20260927 15:07:03.461032 5831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:03.464366 5837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:03.469691 5837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:03.479945 5837 catalog_manager.cc:1347] Generated new cluster ID: 6e4a3aa09f2d4fe0a41a9c8e55152f51
I20260927 15:07:03.480088 5837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:03.503131 5837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:03.504660 5837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:03.516338 5837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472253de659040c2bdaf4abbc8a6b187: Generated new TSK 0
I20260927 15:07:03.517552 5837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:03.553465 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:03.543815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.0.97.62:33245)
0927 15:07:03.544033 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:03.544040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:03.544489 (+ 449us) server_negotiation.cc:407] Connection header received
0927 15:07:03.544737 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:03.544743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:03.544845 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:03.545044 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:03.545493 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:03.546620 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:03.548175 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:03.548964 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:03.549108 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:03.550200 (+ 1092us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:03.552768 (+ 2568us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:07:03.553153 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 15:07:03.553614 5828 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 15:07:03.555330 5855 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:03.543668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33245 (local address 127.0.0.1:44374)
0927 15:07:03.544377 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:03.544394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:03.544538 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:03.545046 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:03.545049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:03.545059 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:03.545324 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:03.545332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:03.546799 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:03.546804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:03.548018 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:03.548031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:03.549257 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:03.549426 (+ 169us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:03.549788 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:03.549858 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:03.553068 (+ 3210us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:07:03.554859 (+ 1791us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1447,"thread_start_us":389,"threads_started":1}
I20260927 15:07:03.556746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5759
I20260927 15:07:03.558054 5820 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 15:07:03.565853 5856 webserver.cc:533] Webserver: sslize failed:
E20260927 15:07:03.565994 5856 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:07:03.652769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5759
2026-09-27T15:07:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (775 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T15:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:03Z Disabled control of system clock
I20260927 15:07:03.739255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43135
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43135
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/testchainca.cert with env {}
W20260927 15:07:04.088445 5861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:04.088781 5861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:04.088953 5861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:04.096565 5861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:04.096697 5861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:04.096763 5861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:04.096858 5861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:04.096922 5861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:04.097004 5861 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:04.097059 5861 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 15:07:04.107640 5861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:04.109201 5861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:04.110630 5861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:04.123469 5866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:04.123580 5867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:04.123772 5869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:04.124859 5861 server_base.cc:1034] running on GCE node
I20260927 15:07:04.125811 5861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:04.127646 5861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:04.128995 5861 hybrid_clock.cc:648] HybridClock initialized: now 1790521624128874 us; error 145 us; skew 500 ppm
I20260927 15:07:04.129467 5861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:04.132786 5861 webserver.cc:460] Webserver started at http://127.0.97.62:40729/ using document root <none> and password file <none>
I20260927 15:07:04.133615 5861 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:04.133783 5861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:04.134132 5861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:04.137367 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "2936fd523058440498270945744382ba"
format_stamp: "Formatted at 2026-09-27 15:07:04 on dist-test-slave-g0w1"
server_key: "29d67ebf81e300022b52983cf35ce992"
server_key_iv: "070b9698a34058892cdbd517d76e1908"
server_key_version: "encryptionkey@0"
I20260927 15:07:04.138317 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "2936fd523058440498270945744382ba"
format_stamp: "Formatted at 2026-09-27 15:07:04 on dist-test-slave-g0w1"
server_key: "29d67ebf81e300022b52983cf35ce992"
server_key_iv: "070b9698a34058892cdbd517d76e1908"
server_key_version: "encryptionkey@0"
I20260927 15:07:04.144351 5861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:07:04.148517 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:04.150285 5861 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:04.150555 5861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "2936fd523058440498270945744382ba"
format_stamp: "Formatted at 2026-09-27 15:07:04 on dist-test-slave-g0w1"
server_key: "29d67ebf81e300022b52983cf35ce992"
server_key_iv: "070b9698a34058892cdbd517d76e1908"
server_key_version: "encryptionkey@0"
I20260927 15:07:04.150918 5861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:04.183439 5861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:04.184366 5861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:04.184641 5861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:04.205600 5861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43135
I20260927 15:07:04.205618 5926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43135 every 8 connection(s)
I20260927 15:07:04.207216 5861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:04.211858 5927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:04.212146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5861
I20260927 15:07:04.212356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:04.213140 388 external_mini_cluster.cc:1428] Setting key 03fc5495abc92a280178b216d976c3b8
I20260927 15:07:04.228379 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba: Bootstrap starting.
I20260927 15:07:04.228597 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:04.215024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.0.97.62:43135)
0927 15:07:04.216157 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:04.216168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:04.216221 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:07:04.217741 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:04.217766 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:04.218300 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:04.218673 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:04.219518 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:04.221002 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:04.223169 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:04.223977 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:04.224212 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:04.225716 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:04.225726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:04.225744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:04.226469 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:04.226562 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:04.226585 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:04.226686 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:04.226952 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 15:07:04.227447 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":355,"threads_started":1}
I20260927 15:07:04.232710 5927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:04.233489 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:04.234022 5927 log.cc:824] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:04.237527 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba: No bootstrap required, opened a new log
I20260927 15:07:04.245942 5927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2936fd523058440498270945744382ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43135 } }
I20260927 15:07:04.246351 5927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:04.246506 5927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2936fd523058440498270945744382ba, State: Initialized, Role: FOLLOWER
I20260927 15:07:04.247390 5927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2936fd523058440498270945744382ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43135 } }
I20260927 15:07:04.247978 5927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:04.248183 5927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:04.248435 5927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:04.253201 5927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2936fd523058440498270945744382ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43135 } }
I20260927 15:07:04.254194 5927 leader_election.cc:302] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2936fd523058440498270945744382ba; no voters:
I20260927 15:07:04.255362 5927 leader_election.cc:288] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:04.255507 5932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:04.256673 5932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [term 1 LEADER]: Becoming Leader. State: Replica: 2936fd523058440498270945744382ba, State: Running, Role: LEADER
I20260927 15:07:04.257354 5932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2936fd523058440498270945744382ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43135 } }
I20260927 15:07:04.258458 5927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:04.262676 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2936fd523058440498270945744382ba. Latest consensus state: current_term: 1 leader_uuid: "2936fd523058440498270945744382ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2936fd523058440498270945744382ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43135 } } }
I20260927 15:07:04.264633 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:04.263206 5933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2936fd523058440498270945744382ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2936fd523058440498270945744382ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43135 } } }
I20260927 15:07:04.265132 5933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:04.265909 5938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:04.274948 5938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:04.298244 5938 catalog_manager.cc:1347] Generated new cluster ID: a4d698379cfe4ae8841cd7e535167324
I20260927 15:07:04.298388 5938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:04.328428 5938 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:04.329986 5938 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:04.345033 5938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2936fd523058440498270945744382ba: Generated new TSK 0
I20260927 15:07:04.346031 5938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:04.496315 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:04.371686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.0.97.62:43135)
0927 15:07:04.371888 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:04.371896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:04.372309 (+ 413us) server_negotiation.cc:407] Connection header received
0927 15:07:04.372546 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:04.372553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:04.372660 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:04.372881 (+ 221us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:04.373392 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:04.374554 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:04.376234 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:04.376950 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:04.377094 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:04.377672 (+ 578us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:04.495524 (+117852us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:07:04.495981 (+ 457us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33840: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:07:04.496480 5930 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33840: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 15:07:04.496465 5956 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:04.371537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43135 (local address 127.0.0.1:33840)
0927 15:07:04.372194 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:04.372211 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:04.372347 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:04.372930 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:04.372934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:04.372946 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:04.373214 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:04.373222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:04.374762 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:04.374767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:04.376051 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:04.376068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:04.377109 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:04.377164 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:04.377295 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:04.377334 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:04.495867 (+118533us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:07:04.496125 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43135: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":334,"threads_started":1}
I20260927 15:07:04.497524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5861
I20260927 15:07:04.498982 5922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:04.602258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5861
2026-09-27T15:07:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (928 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (428 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (433 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:05 dist-test-slave-g0w1 krb5kdc[5971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:05 dist-test-slave-g0w1 krb5kdc[5971](info): set up 2 sockets
Sep 27 15:07:05 dist-test-slave-g0w1 krb5kdc[5971](info): commencing operation
krb5kdc: starting...
W20260927 15:07:06.571834 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:06 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:07:06.799798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:07.143867 5984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:07.144193 5984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:07.144356 5984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:07.151898 5984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:07.152029 5984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:07.152120 5984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:07.152206 5984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:07.152271 5984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:07.162528 5984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:07.163931 5984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:07.165335 5984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:07.177842 5989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:07.177904 5990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:07.177873 5992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:07.178874 5984 server_base.cc:1034] running on GCE node
I20260927 15:07:07.179776 5984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:07.181444 5984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:07.182705 5984 hybrid_clock.cc:648] HybridClock initialized: now 1790521627182651 us; error 50 us; skew 500 ppm
I20260927 15:07:07.183192 5984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:07 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:07.186698 5984 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:07:07.188537 5984 webserver.cc:460] Webserver started at http://127.0.97.62:42405/ using document root <none> and password file <none>
I20260927 15:07:07.189318 5984 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:07.189474 5984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:07.189800 5984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:07.193081 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "10070d7f1021440a8852937ac3c5e556"
format_stamp: "Formatted at 2026-09-27 15:07:07 on dist-test-slave-g0w1"
server_key: "cdd950405a23956f2a7887f400c4fb05"
server_key_iv: "a0c3e1e04350f6e876c8313d0941e794"
server_key_version: "encryptionkey@0"
I20260927 15:07:07.193904 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "10070d7f1021440a8852937ac3c5e556"
format_stamp: "Formatted at 2026-09-27 15:07:07 on dist-test-slave-g0w1"
server_key: "cdd950405a23956f2a7887f400c4fb05"
server_key_iv: "a0c3e1e04350f6e876c8313d0941e794"
server_key_version: "encryptionkey@0"
I20260927 15:07:07.199854 5984 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:07:07.203991 5999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:07.205575 5984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:07.205806 5984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "10070d7f1021440a8852937ac3c5e556"
format_stamp: "Formatted at 2026-09-27 15:07:07 on dist-test-slave-g0w1"
server_key: "cdd950405a23956f2a7887f400c4fb05"
server_key_iv: "a0c3e1e04350f6e876c8313d0941e794"
server_key_version: "encryptionkey@0"
I20260927 15:07:07.206092 5984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:07.223623 5984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:07.228617 5984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:07.228960 5984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:07.249234 5984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37277
I20260927 15:07:07.249256 6050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37277 every 8 connection(s)
I20260927 15:07:07.250696 5984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:07.252187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 5984
I20260927 15:07:07.252411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:07.253197 388 external_mini_cluster.cc:1428] Setting key e7f37a6a7009bf450052adde2aeed12f
I20260927 15:07:07.255684 6051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:07.267452 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556: Bootstrap starting.
Sep 27 15:07:07 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:07.271215 6051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:07.272351 6051 log.cc:824] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:07.275923 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556: No bootstrap required, opened a new log
I20260927 15:07:07.277493 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:07.255401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.97.62:37277)
0927 15:07:07.256256 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:07.256269 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:07.256311 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:07:07.258004 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:07.258029 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:07.258467 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:07.258927 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:07.260425 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:07.261792 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:07.263343 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:07.264098 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:07.264323 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:07.269166 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:07.269175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:07.269191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:07.269236 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:07.272487 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:07.273485 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:07.273493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:07.273499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:07.273703 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:07.274127 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:07.274131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:07.274134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:07.274808 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:07.275139 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:07.275597 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 15:07:07.276113 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":396,"threads_started":1}
I20260927 15:07:07.284798 6051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10070d7f1021440a8852937ac3c5e556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260927 15:07:07.285339 6051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:07.285503 6051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10070d7f1021440a8852937ac3c5e556, State: Initialized, Role: FOLLOWER
I20260927 15:07:07.286126 6051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10070d7f1021440a8852937ac3c5e556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260927 15:07:07.286386 6051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:07.286542 6051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:07.286731 6051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:07.289008 6051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10070d7f1021440a8852937ac3c5e556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260927 15:07:07.289569 6051 leader_election.cc:302] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10070d7f1021440a8852937ac3c5e556; no voters:
I20260927 15:07:07.290277 6051 leader_election.cc:288] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:07.290421 6056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:07.292191 6056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [term 1 LEADER]: Becoming Leader. State: Replica: 10070d7f1021440a8852937ac3c5e556, State: Running, Role: LEADER
I20260927 15:07:07.292730 6051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:07.292829 6056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10070d7f1021440a8852937ac3c5e556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260927 15:07:07.296257 6057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10070d7f1021440a8852937ac3c5e556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10070d7f1021440a8852937ac3c5e556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } } }
I20260927 15:07:07.296308 6058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10070d7f1021440a8852937ac3c5e556. Latest consensus state: current_term: 1 leader_uuid: "10070d7f1021440a8852937ac3c5e556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10070d7f1021440a8852937ac3c5e556" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } } }
I20260927 15:07:07.296742 6057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:07.296749 6058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556 [sys.catalog]: This master's current role is: LEADER
W20260927 15:07:07.303797 6072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:07.303975 6072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:07.304118 6066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:07.305613 6066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:07.315605 6066 catalog_manager.cc:1347] Generated new cluster ID: 67d3ba3bedbd4c08b2513e4692c05580
I20260927 15:07:07.315759 6066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:07.338963 6066 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:07.340711 6066 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:07.349010 6066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10070d7f1021440a8852937ac3c5e556: Generated new TSK 0
I20260927 15:07:07.350174 6066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:07:07.415673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:07.778311 6077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:07.778630 6077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:07.778728 6077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:07.786686 6077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:07.786872 6077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:07.787232 6077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:07.798305 6077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:07.799772 6077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:07.801167 6077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:07.816378 6083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:07.816704 6082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:07.816397 6085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:07.817850 6077 server_base.cc:1034] running on GCE node
I20260927 15:07:07.818564 6077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:07.819685 6077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:07.820920 6077 hybrid_clock.cc:648] HybridClock initialized: now 1790521627820872 us; error 48 us; skew 500 ppm
I20260927 15:07:07.821343 6077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:07 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:07.829118 6077 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:07:07.830936 6077 webserver.cc:460] Webserver started at http://127.0.97.1:33383/ using document root <none> and password file <none>
I20260927 15:07:07.831773 6077 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:07.831892 6077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:07.832147 6077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:07.835404 6077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "557c66699b5c4daf94ae22b4e96b28cb"
format_stamp: "Formatted at 2026-09-27 15:07:07 on dist-test-slave-g0w1"
server_key: "54cc8bf843567b4bc4226785873d9655"
server_key_iv: "cc4b0c61443e004a2c59cbe95ed1bdf6"
server_key_version: "encryptionkey@0"
I20260927 15:07:07.836256 6077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "557c66699b5c4daf94ae22b4e96b28cb"
format_stamp: "Formatted at 2026-09-27 15:07:07 on dist-test-slave-g0w1"
server_key: "54cc8bf843567b4bc4226785873d9655"
server_key_iv: "cc4b0c61443e004a2c59cbe95ed1bdf6"
server_key_version: "encryptionkey@0"
I20260927 15:07:07.842223 6077 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:07:07.846475 6092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:07.848510 6077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:07.848722 6077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "557c66699b5c4daf94ae22b4e96b28cb"
format_stamp: "Formatted at 2026-09-27 15:07:07 on dist-test-slave-g0w1"
server_key: "54cc8bf843567b4bc4226785873d9655"
server_key_iv: "cc4b0c61443e004a2c59cbe95ed1bdf6"
server_key_version: "encryptionkey@0"
I20260927 15:07:07.849016 6077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:07.870189 6077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:07.875603 6077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:07.876044 6077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:07.877257 6077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:07.878794 6077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:07.878880 6077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:07.879027 6077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:07.879145 6077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 15:07:07.915062 6077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38871
I20260927 15:07:07.915153 6204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38871 every 8 connection(s)
I20260927 15:07:07.916797 6077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:07.918419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6077
I20260927 15:07:07.918875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:07.919724 388 external_mini_cluster.cc:1428] Setting key 7ee6a1d2697c5161ee084dafad17bc7f
Sep 27 15:07:07 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:07:07.944500 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:07.921468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50469 (local address 127.0.97.62:37277)
0927 15:07:07.922004 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:07.922014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:07.922277 (+ 263us) server_negotiation.cc:407] Connection header received
0927 15:07:07.925382 (+ 3105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:07.925387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:07.925477 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:07.925620 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:07.928155 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:07.928961 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:07.930745 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:07.931602 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:07.931763 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:07.935988 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:07.935999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:07.936007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:07.936059 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:07.940385 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:07.941432 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:07.941442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:07.941449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:07.941710 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:07.942307 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:07.942315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:07.942321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:07.942783 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:07.942938 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:07.943956 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0927 15:07:07.944221 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":1687,"threads_started":1}
I20260927 15:07:07.945667 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:07.920845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37277 (local address 127.0.97.1:50469)
0927 15:07:07.922074 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:07.922145 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:07.924972 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:07.925962 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:07.925979 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:07.926892 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:07.927884 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:07.927919 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:07.929159 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:07.929165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:07.930576 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:07.930591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:07.931876 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:07.931882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:07.932182 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:07.933006 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:07.933030 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:07.935685 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:07.940628 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:07.940638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:07.940656 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:07.941199 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:07.941867 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:07.941871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:07.941923 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:07.942144 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:07.943015 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:07.943022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:07.943529 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 15:07:07.943818 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 15:07:07.944362 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":404,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:07:07.947585 6205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37277
I20260927 15:07:07.947956 6205 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:07.948945 6205 heartbeater.cc:511] Master 127.0.97.62:37277 requested a full tablet report, sending...
I20260927 15:07:07.951692 6016 ts_manager.cc:194] Registered new tserver with Master: 557c66699b5c4daf94ae22b4e96b28cb (127.0.97.1:38871)
I20260927 15:07:07.953527 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:07:07.978495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:08.328208 6211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:08.328599 6211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:08.328780 6211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:08.337041 6211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:08.337227 6211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:08.337423 6211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:08.347926 6211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:08.349432 6211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:08.350895 6211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:08.365656 6219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:08.365707 6217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:08.365732 6216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:08.366906 6211 server_base.cc:1034] running on GCE node
I20260927 15:07:08.367743 6211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:08.368806 6211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:08.370111 6211 hybrid_clock.cc:648] HybridClock initialized: now 1790521628370041 us; error 54 us; skew 500 ppm
I20260927 15:07:08.370494 6211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:08 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:08.374089 6211 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:07:08.375901 6211 webserver.cc:460] Webserver started at http://127.0.97.2:32969/ using document root <none> and password file <none>
I20260927 15:07:08.376628 6211 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:08.376737 6211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:08.376988 6211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:08.380201 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "c13c0c94b7564ae289ea4628a7e6b4bc"
format_stamp: "Formatted at 2026-09-27 15:07:08 on dist-test-slave-g0w1"
server_key: "5961ec86fa301bb76a917ec2e362a893"
server_key_iv: "752deb8023c701e2d08f4ee04664b699"
server_key_version: "encryptionkey@0"
I20260927 15:07:08.381037 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c13c0c94b7564ae289ea4628a7e6b4bc"
format_stamp: "Formatted at 2026-09-27 15:07:08 on dist-test-slave-g0w1"
server_key: "5961ec86fa301bb76a917ec2e362a893"
server_key_iv: "752deb8023c701e2d08f4ee04664b699"
server_key_version: "encryptionkey@0"
I20260927 15:07:08.387203 6211 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:07:08.391515 6226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:08.393282 6211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:08.393504 6211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "c13c0c94b7564ae289ea4628a7e6b4bc"
format_stamp: "Formatted at 2026-09-27 15:07:08 on dist-test-slave-g0w1"
server_key: "5961ec86fa301bb76a917ec2e362a893"
server_key_iv: "752deb8023c701e2d08f4ee04664b699"
server_key_version: "encryptionkey@0"
I20260927 15:07:08.393729 6211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:08.410686 6211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:08.416157 6211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:08.416530 6211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:08.417709 6211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:08.419366 6211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:08.419448 6211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:08.419624 6211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:08.419683 6211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:08.456480 6211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33893
I20260927 15:07:08.456539 6338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33893 every 8 connection(s)
I20260927 15:07:08.457858 6211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:08.461393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6211
I20260927 15:07:08.461637 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:08.462086 388 external_mini_cluster.cc:1428] Setting key 734bc6acd01a319d40bb54e8c94882b9
Sep 27 15:07:08 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:08.480859 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:08.460766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57877 (local address 127.0.97.62:37277)
0927 15:07:08.461357 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:08.461365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:08.462617 (+ 1252us) server_negotiation.cc:407] Connection header received
0927 15:07:08.464811 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:08.464816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:08.464911 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:08.465042 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:08.466749 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:08.467582 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:08.469287 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:08.469718 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:08.469864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:08.474035 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:08.474041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:08.474044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:08.474080 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:08.477213 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:08.478132 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:08.478139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:08.478145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:08.478293 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:08.478817 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:08.478828 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:08.478830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:08.479207 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:08.479376 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:08.480162 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 15:07:08.480425 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":314,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:07:08.482123 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:08.461425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37277 (local address 127.0.97.2:57877)
0927 15:07:08.462418 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:08.462468 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:08.464496 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:08.465317 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:08.465326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:08.465882 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:08.466547 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:08.466574 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:08.467771 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:08.467775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:08.469106 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:08.469120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:08.469970 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:08.469977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:08.470255 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:08.471143 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:08.471188 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:08.473731 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:08.477416 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:08.477427 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:08.477450 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:08.477962 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:08.478427 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:08.478433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:08.478437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:08.478672 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:08.479350 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:08.479360 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:08.479763 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 15:07:08.480078 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 15:07:08.480708 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":428,"threads_started":1}
I20260927 15:07:08.483978 6339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:07:08.484403 6339 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:08.485247 6339 heartbeater.cc:511] Master 127.0.97.62:37277 requested a full tablet report, sending...
I20260927 15:07:08.486901 6016 ts_manager.cc:194] Registered new tserver with Master: c13c0c94b7564ae289ea4628a7e6b4bc (127.0.97.2:33893)
I20260927 15:07:08.487879 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:07:08.512682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:08.866845 6345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:08.867199 6345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:08.867298 6345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:08.874748 6345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:08.874887 6345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:08.875007 6345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:08.885596 6345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:08.886961 6345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:08.888384 6345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:08.902886 6351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:08.902947 6353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:08.902995 6350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:08.903980 6345 server_base.cc:1034] running on GCE node
I20260927 15:07:08.904706 6345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:08.905835 6345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:08.907150 6345 hybrid_clock.cc:648] HybridClock initialized: now 1790521628907031 us; error 36 us; skew 500 ppm
I20260927 15:07:08.907559 6345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:08 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:08.911468 6345 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:07:08.913231 6345 webserver.cc:460] Webserver started at http://127.0.97.3:35261/ using document root <none> and password file <none>
I20260927 15:07:08.914000 6345 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:08.914214 6345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:08.914608 6345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:08.919137 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "81bf38ae5cdb409f86c17428ac5accb9"
format_stamp: "Formatted at 2026-09-27 15:07:08 on dist-test-slave-g0w1"
server_key: "b17b09ced4b392572dcd896d6d33b67f"
server_key_iv: "dfda204750d976283395ca91000bef54"
server_key_version: "encryptionkey@0"
I20260927 15:07:08.920351 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "81bf38ae5cdb409f86c17428ac5accb9"
format_stamp: "Formatted at 2026-09-27 15:07:08 on dist-test-slave-g0w1"
server_key: "b17b09ced4b392572dcd896d6d33b67f"
server_key_iv: "dfda204750d976283395ca91000bef54"
server_key_version: "encryptionkey@0"
I20260927 15:07:08.926973 6345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:08.930958 6360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:08.932716 6345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:07:08.932950 6345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "81bf38ae5cdb409f86c17428ac5accb9"
format_stamp: "Formatted at 2026-09-27 15:07:08 on dist-test-slave-g0w1"
server_key: "b17b09ced4b392572dcd896d6d33b67f"
server_key_iv: "dfda204750d976283395ca91000bef54"
server_key_version: "encryptionkey@0"
I20260927 15:07:08.933197 6345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:08.955889 6345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:08.957340 6205 heartbeater.cc:503] Master 127.0.97.62:37277 was elected leader, sending a full tablet report...
I20260927 15:07:08.961031 6345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:08.961360 6345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:08.962432 6345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:08.964018 6345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:08.964120 6345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:08.964242 6345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:08.964303 6345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:09.000515 6345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38511
I20260927 15:07:09.000612 6472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38511 every 8 connection(s)
I20260927 15:07:09.001901 6345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:09.005645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6345
I20260927 15:07:09.005851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:09.006482 388 external_mini_cluster.cc:1428] Setting key 9b5123e4fe99b87d07e7a34747199c55
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:09.024674 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.004954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32851 (local address 127.0.97.62:37277)
0927 15:07:09.005564 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.005569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.007164 (+ 1595us) server_negotiation.cc:407] Connection header received
0927 15:07:09.008826 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.008832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.008946 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.009135 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.011195 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.012123 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.013785 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.014238 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.014348 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.018098 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.018104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.018107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.018144 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.021290 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.022140 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.022148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.022151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.022301 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.022755 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.022761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.022765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.023044 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.023254 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.024018 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.024259 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":376,"threads_started":1}
I20260927 15:07:09.025693 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.005559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37277 (local address 127.0.97.3:32851)
0927 15:07:09.006414 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:09.006485 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:09.008473 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:09.009483 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:09.009498 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:09.010148 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:09.010880 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:09.010903 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.012299 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.012303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:09.013630 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:09.013645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.014451 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.014478 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:09.014703 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.015477 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:09.015508 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:09.017834 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:09.021457 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:09.021469 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:09.021489 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:09.021995 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:09.022421 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:09.022426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:09.022430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:09.022641 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:09.023228 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:09.023237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:09.023631 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 15:07:09.023929 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 15:07:09.024526 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":387,"threads_started":1}
I20260927 15:07:09.027438 6473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37277
I20260927 15:07:09.027812 6473 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:09.028614 6473 heartbeater.cc:511] Master 127.0.97.62:37277 requested a full tablet report, sending...
I20260927 15:07:09.030340 6016 ts_manager.cc:194] Registered new tserver with Master: 81bf38ae5cdb409f86c17428ac5accb9 (127.0.97.3:38511)
I20260927 15:07:09.031265 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32851
I20260927 15:07:09.035832 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/test-admin.keytab.
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.064178 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 15:07:09.070043 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:09.067042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.0.97.1:38871)
0927 15:07:09.067901 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.067910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.067944 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:07:09.068134 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.068141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.068362 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.068589 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.069613 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37440: BlockingRecv error: recv got EOF from 127.0.0.1:37440 (error 108)
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":485,"threads_started":1}
W20260927 15:07:09.070319 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.072247 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.073698 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:07:09.090520 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.075630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.97.1:38871)
0927 15:07:09.075873 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.075880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.076159 (+ 279us) server_negotiation.cc:407] Connection header received
0927 15:07:09.076298 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.076302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.076367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.076457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.077516 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.078547 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.079947 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.080563 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.080669 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.084575 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.084591 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.084607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.084655 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.087520 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.088307 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.088315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.088318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.088462 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.088855 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.088859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.088861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.089313 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.089543 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.089832 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.090197 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260927 15:07:09.097150 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:09.095156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.0.97.1:38871)
0927 15:07:09.095365 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.095372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.095604 (+ 232us) server_negotiation.cc:407] Connection header received
0927 15:07:09.095802 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.095807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.095887 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.095996 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.096874 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37452: BlockingRecv error: recv got EOF from 127.0.0.1:37452 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 15:07:09.097381 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.099277 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.100746 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:07:09.116559 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.102397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.0.97.1:38871)
0927 15:07:09.102585 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.102590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.103153 (+ 563us) server_negotiation.cc:407] Connection header received
0927 15:07:09.103315 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.103320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.103385 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.103475 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.104466 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.105231 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.106518 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.107047 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.107297 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.111464 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.111470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.111474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.111522 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.114073 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.114735 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.114739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.114741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.114863 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.115281 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.115286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.115288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.115581 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.115795 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.116052 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.116240 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:07:09.121270 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:09.119428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.0.97.1:38871)
0927 15:07:09.119594 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.119599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.119985 (+ 386us) server_negotiation.cc:407] Connection header received
0927 15:07:09.120138 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.120142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.120212 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.120289 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.121002 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37464: BlockingRecv error: recv got EOF from 127.0.0.1:37464 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:07:09.121472 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521577514430-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.123299 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:09.124845 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:07:09.141884 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.126336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.0.97.1:38871)
0927 15:07:09.126611 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.126618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.126834 (+ 216us) server_negotiation.cc:407] Connection header received
0927 15:07:09.126978 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.126983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.127044 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.127199 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.128169 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.128901 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.130474 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.130932 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.131045 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.135446 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.135452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.135455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.135497 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.138566 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.139619 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.139624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.139627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.139824 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.140287 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.140292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.140295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.140618 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.140883 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.141229 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.141496 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:07:09.143905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6077
I20260927 15:07:09.157122 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.144709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.0.97.1:38871)
0927 15:07:09.144953 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.144962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.145267 (+ 305us) server_negotiation.cc:407] Connection header received
0927 15:07:09.145439 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.145444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.145513 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.145605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.146718 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.147506 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.148774 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.149227 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.149335 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.151676 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.151682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.151686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.151721 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.154469 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.155206 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.155210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.155213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.155340 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.155690 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.155693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.155695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.155936 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.156177 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.156515 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.156689 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:07:09.157768 6200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:09.244184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6077
I20260927 15:07:09.261248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6211
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:07:09.278249 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.262382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.0.97.2:33893)
0927 15:07:09.262987 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.262994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.263135 (+ 141us) server_negotiation.cc:407] Connection header received
0927 15:07:09.263222 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.263228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.263454 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.263602 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.264545 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.265598 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.266963 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.267812 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.267934 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.272011 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.272027 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.272043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.272085 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.275196 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.275943 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.275951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.275954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.276137 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.276644 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.276648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.276651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.277020 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.277271 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.277603 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.277915 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
I20260927 15:07:09.279641 6334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:09.363538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6211
I20260927 15:07:09.380805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6345
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:07:09.399178 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:09.382005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.0.97.3:38511)
0927 15:07:09.382755 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:09.382762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:09.382792 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:07:09.382851 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:09.382858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:09.383170 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:09.383325 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:09.384577 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.385616 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.387144 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:09.387744 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:09.387848 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:09.392593 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:09.392608 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:09.392624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:09.392669 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:09.396104 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.396909 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.396918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.396921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.397058 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:09.397486 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:09.397491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:09.397496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:09.397869 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:09.398189 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:09.398529 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:07:09.398812 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":428,"threads_started":1}
I20260927 15:07:09.400554 6468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:09.486865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6345
I20260927 15:07:09.502980 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 5984
I20260927 15:07:09.504379 6046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:09.609602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 5984
2026-09-27T15:07:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4151 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[6501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[6501](info): set up 2 sockets
Sep 27 15:07:09 dist-test-slave-g0w1 krb5kdc[6501](info): commencing operation
krb5kdc: starting...
W20260927 15:07:10.731925 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:10 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 15:07:10.963543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34775
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:11.298899 6514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:11.299305 6514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:11.299471 6514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:11.306978 6514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:11.307201 6514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:11.307317 6514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:11.307418 6514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:11.307490 6514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:11.318253 6514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:11.319849 6514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:11.321295 6514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:11.334379 6522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:11.334371 6520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:11.334786 6519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:11.335676 6514 server_base.cc:1034] running on GCE node
I20260927 15:07:11.336642 6514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:11.338369 6514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:11.339658 6514 hybrid_clock.cc:648] HybridClock initialized: now 1790521631339615 us; error 41 us; skew 500 ppm
I20260927 15:07:11.340093 6514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:11 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521631, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:11.343719 6514 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 15:07:11.345387 6514 webserver.cc:460] Webserver started at http://127.0.97.62:34191/ using document root <none> and password file <none>
I20260927 15:07:11.346154 6514 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:11.346306 6514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:11.346621 6514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:11.349892 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "54375309f80f4087988e32e540f70f37"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "260e898780c1794d937e47c3d228209f"
server_key_iv: "5b0f3d7f8c8dbe10fd4aedd12ee6d88c"
server_key_version: "encryptionkey@0"
I20260927 15:07:11.350780 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "54375309f80f4087988e32e540f70f37"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "260e898780c1794d937e47c3d228209f"
server_key_iv: "5b0f3d7f8c8dbe10fd4aedd12ee6d88c"
server_key_version: "encryptionkey@0"
I20260927 15:07:11.356629 6514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:11.360536 6529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:11.362140 6514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:11.362340 6514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "54375309f80f4087988e32e540f70f37"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "260e898780c1794d937e47c3d228209f"
server_key_iv: "5b0f3d7f8c8dbe10fd4aedd12ee6d88c"
server_key_version: "encryptionkey@0"
I20260927 15:07:11.362551 6514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:11.388363 6514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:11.393014 6514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:11.393342 6514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:11.413239 6514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34775
I20260927 15:07:11.413290 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34775 every 8 connection(s)
I20260927 15:07:11.414686 6514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:11.415483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6514
I20260927 15:07:11.415747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:11.416561 388 external_mini_cluster.cc:1428] Setting key 0c24a3adaaeb5367b9546de9f8020ab5
I20260927 15:07:11.419024 6581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:11.433238 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Bootstrap starting.
Sep 27 15:07:11 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:11.437402 6581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:11.438844 6581 log.cc:824] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:11.442953 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: No bootstrap required, opened a new log
I20260927 15:07:11.444594 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:11.418932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.0.97.62:34775)
0927 15:07:11.419956 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:11.419972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:11.420460 (+ 488us) server_negotiation.cc:407] Connection header received
0927 15:07:11.422494 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:11.422526 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:11.423181 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:11.423699 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:11.425111 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:11.427270 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:11.428777 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:11.429898 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:11.430121 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:11.434770 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:11.434788 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:11.434815 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:11.434883 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:11.439046 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:11.440201 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:11.440210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:11.440217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:11.440496 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:11.440955 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:11.440960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:11.440964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:11.441765 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:11.442170 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:11.442469 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 15:07:11.442988 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":457,"threads_started":1}
I20260927 15:07:11.452703 6581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } }
I20260927 15:07:11.453110 6581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:11.453258 6581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54375309f80f4087988e32e540f70f37, State: Initialized, Role: FOLLOWER
I20260927 15:07:11.453892 6581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } }
I20260927 15:07:11.454159 6581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:11.454319 6581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:11.454504 6581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:11.457039 6581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } }
I20260927 15:07:11.457692 6581 leader_election.cc:302] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54375309f80f4087988e32e540f70f37; no voters:
I20260927 15:07:11.458269 6581 leader_election.cc:288] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:11.458427 6586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:11.459156 6586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 LEADER]: Becoming Leader. State: Replica: 54375309f80f4087988e32e540f70f37, State: Running, Role: LEADER
I20260927 15:07:11.459756 6586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } }
I20260927 15:07:11.460608 6581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:11.462826 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } }
I20260927 15:07:11.463197 6588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54375309f80f4087988e32e540f70f37. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } }
I20260927 15:07:11.463394 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:11.463539 6588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:11.465545 6595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:11.471328 6595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:11.480994 6595 catalog_manager.cc:1347] Generated new cluster ID: 147140b2f02b47deaa8e84253542175e
I20260927 15:07:11.481144 6595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:11.496371 6595 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:11.497823 6595 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:11.509020 6595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Generated new TSK 0
I20260927 15:07:11.510057 6595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 15:07:11.566833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34775
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:11.940025 6607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:11.940330 6607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:11.940455 6607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:11.948287 6607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:11.948415 6607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:11.948532 6607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:11.961534 6607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:11.963131 6607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:11.965082 6607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:11.979311 6615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:11.979427 6613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:11.979465 6612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:11.980118 6607 server_base.cc:1034] running on GCE node
I20260927 15:07:11.980808 6607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:11.981837 6607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:11.983031 6607 hybrid_clock.cc:648] HybridClock initialized: now 1790521631982973 us; error 70 us; skew 500 ppm
I20260927 15:07:11.983435 6607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:11 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521631, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:11.986665 6607 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 15:07:11.988354 6607 webserver.cc:460] Webserver started at http://127.0.97.1:42721/ using document root <none> and password file <none>
I20260927 15:07:11.989122 6607 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:11.989274 6607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:11.989583 6607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:11.992600 6607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "d11403f429944d3d850732cf48ea6fff"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "35ca544a095095b446127263595df7d0"
server_key_iv: "f4ffb76d03b3a8cf0c2d389da7e16b43"
server_key_version: "encryptionkey@0"
I20260927 15:07:11.993414 6607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d11403f429944d3d850732cf48ea6fff"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "35ca544a095095b446127263595df7d0"
server_key_iv: "f4ffb76d03b3a8cf0c2d389da7e16b43"
server_key_version: "encryptionkey@0"
I20260927 15:07:11.998766 6607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:07:12.002348 6622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:12.003850 6607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:12.004032 6607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "d11403f429944d3d850732cf48ea6fff"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "35ca544a095095b446127263595df7d0"
server_key_iv: "f4ffb76d03b3a8cf0c2d389da7e16b43"
server_key_version: "encryptionkey@0"
I20260927 15:07:12.004257 6607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:12.018492 6607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:12.023025 6607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:12.023406 6607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:12.024411 6607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:12.025792 6607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:12.025897 6607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:12.026051 6607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:12.026124 6607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:12.061671 6607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41133
I20260927 15:07:12.061764 6734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41133 every 8 connection(s)
I20260927 15:07:12.063190 6607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:12.073268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6607
I20260927 15:07:12.073530 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:12.074173 388 external_mini_cluster.cc:1428] Setting key 1fe07e60237abf9e6c3858497377ddfa
Sep 27 15:07:12 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521631, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:12.085601 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:12.066221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59685 (local address 127.0.97.62:34775)
0927 15:07:12.066707 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:12.066713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:12.068125 (+ 1412us) server_negotiation.cc:407] Connection header received
0927 15:07:12.070021 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:12.070026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:12.070099 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:12.070206 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:12.072026 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.072761 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.074638 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.075172 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.075277 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:12.079586 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:12.079591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:12.079595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:12.079631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:12.082308 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:12.083182 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:12.083187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:12.083190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:12.083309 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:12.083776 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:12.083783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:12.083791 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:12.084067 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:12.084238 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:12.085001 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 15:07:12.085239 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260927 15:07:12.086661 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:12.066878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:59685)
0927 15:07:12.067899 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:12.067947 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:12.069683 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:12.070443 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:12.070452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:12.070985 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:12.071772 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:12.071791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.072935 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.072939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:12.074459 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:12.074472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.075394 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.075400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:12.075616 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:12.076462 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:12.076497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:12.079332 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:12.082506 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:12.082517 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:12.082536 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:12.082999 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:12.083431 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:12.083436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:12.083440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:12.083651 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:12.084200 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:12.084211 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:12.084606 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 15:07:12.084884 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 15:07:12.085471 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":375,"threads_started":1}
I20260927 15:07:12.088411 6735 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34775
I20260927 15:07:12.088778 6735 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:12.089764 6735 heartbeater.cc:511] Master 127.0.97.62:34775 requested a full tablet report, sending...
I20260927 15:07:12.092479 6546 ts_manager.cc:194] Registered new tserver with Master: d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133)
I20260927 15:07:12.094715 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59685
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 15:07:12.123490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34775
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:12.458657 6741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:12.459069 6741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:12.459275 6741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:12.467614 6741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:12.467770 6741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:12.467947 6741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:12.478432 6741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:12.479869 6741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:12.481240 6741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:12.496134 6746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:12.496212 6747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:12.496184 6749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:12.496708 6741 server_base.cc:1034] running on GCE node
I20260927 15:07:12.497502 6741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:12.498700 6741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:12.499945 6741 hybrid_clock.cc:648] HybridClock initialized: now 1790521632499851 us; error 87 us; skew 500 ppm
I20260927 15:07:12.500308 6741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:12 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521632, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:12.503777 6741 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 15:07:12.505419 6741 webserver.cc:460] Webserver started at http://127.0.97.2:45185/ using document root <none> and password file <none>
I20260927 15:07:12.506191 6741 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:12.506330 6741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:12.506657 6741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:12.510239 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "dc9c744a346d4c3fa599ef02298835ae"
format_stamp: "Formatted at 2026-09-27 15:07:12 on dist-test-slave-g0w1"
server_key: "96e0ec78cc473fa3d78ff128980d29c3"
server_key_iv: "b38045bb4c9eaf941c0796800438269e"
server_key_version: "encryptionkey@0"
I20260927 15:07:12.511200 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dc9c744a346d4c3fa599ef02298835ae"
format_stamp: "Formatted at 2026-09-27 15:07:12 on dist-test-slave-g0w1"
server_key: "96e0ec78cc473fa3d78ff128980d29c3"
server_key_iv: "b38045bb4c9eaf941c0796800438269e"
server_key_version: "encryptionkey@0"
I20260927 15:07:12.516949 6741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:12.521242 6756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:12.522974 6741 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 15:07:12.523290 6741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "dc9c744a346d4c3fa599ef02298835ae"
format_stamp: "Formatted at 2026-09-27 15:07:12 on dist-test-slave-g0w1"
server_key: "96e0ec78cc473fa3d78ff128980d29c3"
server_key_iv: "b38045bb4c9eaf941c0796800438269e"
server_key_version: "encryptionkey@0"
I20260927 15:07:12.523553 6741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:12.550114 6741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:12.554771 6741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:12.555122 6741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:12.556156 6741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:12.557626 6741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:12.557737 6741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:12.557895 6741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:12.557969 6741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:12.594072 6741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42201
I20260927 15:07:12.594182 6868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42201 every 8 connection(s)
I20260927 15:07:12.595531 6741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:12.605289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6741
I20260927 15:07:12.605501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:12.606040 388 external_mini_cluster.cc:1428] Setting key bccac652e66d1589fda5db02b22703e9
Sep 27 15:07:12 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521632, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:12.620420 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:12.598370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52005 (local address 127.0.97.62:34775)
0927 15:07:12.599023 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:12.599028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:12.600026 (+ 998us) server_negotiation.cc:407] Connection header received
0927 15:07:12.601986 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:12.601991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:12.602084 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:12.602218 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:12.603918 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.604655 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.607049 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.607605 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.607791 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:12.612563 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:12.612573 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:12.612581 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:12.612644 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:12.616228 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:12.617533 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:12.617539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:12.617542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:12.617697 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:12.618354 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:12.618358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:12.618360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:12.618654 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:12.618774 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:12.619880 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0927 15:07:12.620132 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":298,"threads_started":1}
I20260927 15:07:12.621968 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:12.599015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:52005)
0927 15:07:12.599846 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:12.599887 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:12.601677 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:12.602471 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:12.602480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:12.603055 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:12.603727 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:12.603745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.604829 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.604854 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:12.606881 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:12.606895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:12.607852 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:12.607863 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:12.608148 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:12.609085 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:12.609129 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:12.612229 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:12.616453 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:12.616467 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:12.616491 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:12.617322 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:12.617833 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:12.617842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:12.617848 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:12.618198 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:12.618804 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:12.618815 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:12.619389 (+ 574us) client_negotiation.cc:769] Sending connection context
0927 15:07:12.619736 (+ 347us) client_negotiation.cc:240] Negotiation successful
0927 15:07:12.620371 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":301,"threads_started":1}
I20260927 15:07:12.623852 6869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34775
I20260927 15:07:12.624238 6869 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:12.625058 6869 heartbeater.cc:511] Master 127.0.97.62:34775 requested a full tablet report, sending...
I20260927 15:07:12.626796 6546 ts_manager.cc:194] Registered new tserver with Master: dc9c744a346d4c3fa599ef02298835ae (127.0.97.2:42201)
I20260927 15:07:12.628224 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52005
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 15:07:12.662408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34775
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:13.010267 6875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:13.010584 6875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:13.010744 6875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:13.018235 6875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:13.018361 6875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:13.018553 6875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:13.029067 6875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:13.030464 6875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:13.031888 6875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:13.046016 6881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:13.046052 6880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:13.046095 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:13.047148 6875 server_base.cc:1034] running on GCE node
I20260927 15:07:13.047856 6875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:13.048969 6875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:13.050254 6875 hybrid_clock.cc:648] HybridClock initialized: now 1790521633050206 us; error 46 us; skew 500 ppm
I20260927 15:07:13.050662 6875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:13.059158 6875 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 15:07:13.060947 6875 webserver.cc:460] Webserver started at http://127.0.97.3:46191/ using document root <none> and password file <none>
I20260927 15:07:13.061779 6875 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:13.061935 6875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:13.062256 6875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:13.065451 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "2a8748a6950c4056bd5e326e4c250701"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "e6eea981564d90585d83efbbf05006a0"
server_key_iv: "cb94797692b4ab4d765531363ed11f84"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.066377 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2a8748a6950c4056bd5e326e4c250701"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "e6eea981564d90585d83efbbf05006a0"
server_key_iv: "cb94797692b4ab4d765531363ed11f84"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.072194 6875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:07:13.076056 6890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:13.077432 6875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:07:13.077631 6875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "2a8748a6950c4056bd5e326e4c250701"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "e6eea981564d90585d83efbbf05006a0"
server_key_iv: "cb94797692b4ab4d765531363ed11f84"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.077853 6875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:13.098234 6735 heartbeater.cc:503] Master 127.0.97.62:34775 was elected leader, sending a full tablet report...
I20260927 15:07:13.111375 6875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:13.116169 6875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:13.116487 6875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:13.117532 6875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:13.118945 6875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:13.119088 6875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:13.119246 6875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:13.119320 6875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:13.155187 6875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43795
I20260927 15:07:13.155256 7002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43795 every 8 connection(s)
I20260927 15:07:13.156533 6875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:13.165563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 6875
I20260927 15:07:13.165774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:13.166210 388 external_mini_cluster.cc:1428] Setting key ccc483ab7c67ba7277a9c591da7a2c8a
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:13.179327 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.159335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48231 (local address 127.0.97.62:34775)
0927 15:07:13.159781 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.159786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.160889 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 15:07:13.162868 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.162873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.162991 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.163161 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.164943 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.165917 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.168331 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.168979 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.169158 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.172939 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.172947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.172951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.172986 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.176117 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.177061 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.177066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.177068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.177199 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.177633 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.177636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.177638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.177898 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.178051 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.178748 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.178926 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":239,"threads_started":1}
I20260927 15:07:13.180431 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.159879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:48231)
0927 15:07:13.160679 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.160730 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.162549 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.163411 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.163420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.164036 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.164762 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.164783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.166110 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.166118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.168134 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.168151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.169312 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.169322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.169595 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.170291 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.170319 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.172680 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.176332 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.176341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.176357 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.176856 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.177319 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.177322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.177324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.177522 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.178008 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.178014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.178374 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.178616 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.179035 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":297,"threads_started":1}
I20260927 15:07:13.182013 7003 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34775
I20260927 15:07:13.182377 7003 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:13.183240 7003 heartbeater.cc:511] Master 127.0.97.62:34775 requested a full tablet report, sending...
I20260927 15:07:13.184713 6546 ts_manager.cc:194] Registered new tserver with Master: 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795)
I20260927 15:07:13.185627 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48231
I20260927 15:07:13.194962 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 15:07:13.243590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34685
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34775,127.0.97.61:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:13.582643 7009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:13.583067 7009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:13.583343 7009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:13.590863 7009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:13.591033 7009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:13.591152 7009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:13.591240 7009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:13.591306 7009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:13.602198 7009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34775,127.0.97.61:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
W20260927 15:07:13.602646 7009 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:34775,127.0.97.61:34685'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 15:07:13.603857 7009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:13.605268 7009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:13.618436 7017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:13.618537 7015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:13.618465 7014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:13.625560 7009 server_base.cc:1034] running on GCE node
I20260927 15:07:13.626570 7009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:13.628443 7009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:13.629760 7009 hybrid_clock.cc:648] HybridClock initialized: now 1790521633629711 us; error 73 us; skew 500 ppm
I20260927 15:07:13.630165 7009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:13.631313 6869 heartbeater.cc:503] Master 127.0.97.62:34775 was elected leader, sending a full tablet report...
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:13.633901 7009 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 15:07:13.635819 7009 webserver.cc:460] Webserver started at http://127.0.97.61:41431/ using document root <none> and password file <none>
I20260927 15:07:13.636673 7009 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:13.636775 7009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:13.637113 7009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:13.640435 7009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/instance:
uuid: "02ff713ce03f4875a4aaf3b237c52857"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "63872ceacb893aa7fccf9d67bfd2394c"
server_key_iv: "3d709d259c627b13b14912b1e653748e"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.641419 7009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal/instance:
uuid: "02ff713ce03f4875a4aaf3b237c52857"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "63872ceacb893aa7fccf9d67bfd2394c"
server_key_iv: "3d709d259c627b13b14912b1e653748e"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.647575 7009 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:07:13.651690 7024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:13.653472 7009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:13.653690 7009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
uuid: "02ff713ce03f4875a4aaf3b237c52857"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "63872ceacb893aa7fccf9d67bfd2394c"
server_key_iv: "3d709d259c627b13b14912b1e653748e"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.653942 7009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:13.680321 7009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:13.685865 7009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:13.686189 7009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:13.704830 7009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34685
I20260927 15:07:13.704839 7075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34685 every 8 connection(s)
I20260927 15:07:13.706341 7009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
I20260927 15:07:13.710659 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:13.715813 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 7009
I20260927 15:07:13.718096 7076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 15:07:13.738412 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.721542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.0.97.62:34775)
0927 15:07:13.722072 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.722079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.722839 (+ 760us) server_negotiation.cc:407] Connection header received
0927 15:07:13.722915 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.722918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.722991 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.723148 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.724419 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.725204 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.726773 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.727237 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.727374 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.731247 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.731253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.731256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.731293 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.734468 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.735541 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.735545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.735547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.735691 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.736184 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.736190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.736194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.736654 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.736789 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.737697 (+ 908us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.738035 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260927 15:07:13.739694 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.721542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46320)
0927 15:07:13.722518 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.722570 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.722729 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.723219 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.723227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.723869 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.724243 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.724257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.725352 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.725358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.726612 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.726624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.727550 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.727560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.727691 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.728502 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.728524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.730984 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.734640 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.734665 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.734679 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.735405 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.735812 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.735817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.735820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.736061 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.736778 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.736787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.737252 (+ 465us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.737612 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.738114 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":570,"threads_started":1}
I20260927 15:07:13.739715 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.718439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.0.97.61:34685)
0927 15:07:13.719356 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.719372 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.719429 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:07:13.721689 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.721747 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.722324 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.722712 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.724252 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.725444 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.726694 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.727317 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.727589 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.731961 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.731972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.731989 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.732055 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.735858 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.736736 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.736746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.736750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.736902 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.737271 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.737279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.737285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.737693 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.737938 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.738342 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.738564 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":368,"threads_started":1}
I20260927 15:07:13.741784 7076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 15:07:13.754545 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.742562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55510)
0927 15:07:13.742780 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.742802 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.742916 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.743652 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.743656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.743964 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.744092 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.744099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.745012 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.745016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.745887 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.745897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.746569 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.746573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.746737 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.747382 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.747399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.749413 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.752397 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.752401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.752404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.752786 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.753152 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.753155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.753157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.753275 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.753775 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.753778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.753905 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.754016 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.754248 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 15:07:13.754642 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.743252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.0.97.61:34685)
0927 15:07:13.743407 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.743417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.743430 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:13.743468 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.743471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.743523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.743601 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.744215 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.744906 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.746007 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.746441 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.746544 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.749611 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.749616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.749620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.749648 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.752258 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.752903 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.752907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.752910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.753031 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.753365 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.753368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.753370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.753675 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.753781 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.754171 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.754366 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:07:13.759543 7076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Bootstrap starting.
I20260927 15:07:13.762737 7076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:13.763823 7076 log.cc:824] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:13.766793 7076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: No bootstrap required, opened a new log
I20260927 15:07:13.773872 7076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } }
I20260927 15:07:13.774219 7076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:13.774286 7076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ff713ce03f4875a4aaf3b237c52857, State: Initialized, Role: FOLLOWER
I20260927 15:07:13.774821 7076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } }
I20260927 15:07:13.775789 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } } }
I20260927 15:07:13.776152 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:13.777050 7076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:07:13.786855 7094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:13.787025 7094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:13.788594 7009 master_runner.cc:428] Detected that this master 02ff713ce03f4875a4aaf3b237c52857 is joining an existing cluster
I20260927 15:07:13.788729 7009 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:34685
I20260927 15:07:13.821350 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.793611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.97.61:34685)
0927 15:07:13.793772 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.793779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.802551 (+ 8772us) server_negotiation.cc:407] Connection header received
0927 15:07:13.802764 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.802768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.802836 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.802901 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.804661 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.805790 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.808130 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.809056 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.809161 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.813679 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.813686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.813692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.813732 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.817900 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.818818 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.818826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.818833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.818998 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.819447 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.819455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.819461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.819852 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.820000 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.820533 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.820838 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:07:13.821419 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.793032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.0.97.62:34775)
0927 15:07:13.793259 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.793268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.802183 (+ 8915us) server_negotiation.cc:407] Connection header received
0927 15:07:13.802529 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.802539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.802644 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.802726 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.804677 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.805783 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.808237 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.808949 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.809111 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.812712 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.812731 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.812738 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.812787 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.816599 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.817640 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.817647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.817650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.817819 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.818279 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.818285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.818289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.818723 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.818876 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.820816 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.821091 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:07:13.822300 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.793084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46328)
0927 15:07:13.802003 (+ 8919us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.802028 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.802335 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.802939 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.802944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.803882 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.804490 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.804499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.805975 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.805979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.808016 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.808034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.809455 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.809463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.809655 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.810339 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.810369 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.811752 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.816820 (+ 5068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.816826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.816830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.817462 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.817962 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.817967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.817971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.818140 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.818939 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.818944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.820281 (+ 1337us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.820474 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.821795 (+ 1321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8748,"mutex_wait_us":1076,"spinlock_wait_cycles":41856,"thread_start_us":9068,"threads_started":1}
I20260927 15:07:13.823114 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.793144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55512)
0927 15:07:13.802475 (+ 9331us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.802492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.802620 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.803100 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.803107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.803882 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.804478 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.804493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.806068 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.806078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.807973 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.807992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.809543 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.809552 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.809833 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.810573 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.810599 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.813479 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.818059 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.818066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.818073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.818697 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.819167 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.819174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.819180 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.819341 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.820067 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.820075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.820243 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.821921 (+ 1678us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.822277 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9177,"mutex_wait_us":95,"spinlock_wait_cycles":36736,"thread_start_us":8507,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 15:07:13 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521631, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 15:07:13.850883 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.829533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.0.97.61:34685)
0927 15:07:13.829720 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.829726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.831862 (+ 2136us) server_negotiation.cc:407] Connection header received
0927 15:07:13.832287 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.832295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.832427 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.832542 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.834098 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.835187 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.836981 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.837888 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.838026 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.843123 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.843133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.843141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.843190 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.847521 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.848579 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.848587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.848593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.848790 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.849255 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.849261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.849266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.849686 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.849861 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.850429 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.850615 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:13.851209 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.829538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55520)
0927 15:07:13.831706 (+ 2168us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.831759 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.832024 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.832622 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.832631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.833586 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.833905 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.833920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.835333 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.835339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.836786 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.836806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.838278 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.838286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.838540 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.839465 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.839495 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.842820 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.847741 (+ 4921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.847760 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.847770 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.848420 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.848941 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.848946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.848950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.849120 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.849853 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.849865 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.850089 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.850310 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.850683 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1883,"thread_start_us":1703,"threads_started":1}
I20260927 15:07:13.853312 6546 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:34685: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 34685
}
attrs {
promote: true
}
}
dest_uuid: "54375309f80f4087988e32e540f70f37"
cas_config_opid_index: -1
I20260927 15:07:13.854637 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 15:07:13.859320 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33959
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:13.859809 7051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:13.865149 7051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Refusing update from remote peer 54375309f80f4087988e32e540f70f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 15:07:13.864562 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02ff713ce03f4875a4aaf3b237c52857 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:13.867753 7106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:13.869978 7107 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:34685
I20260927 15:07:13.870450 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:13.870819 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:13.870869 7009 master.cc:537] Master@127.0.97.61:34685 shutting down...
I20260927 15:07:13.893327 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02ff713ce03f4875a4aaf3b237c52857 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:13.912171 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54375309f80f4087988e32e540f70f37. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:13.912128 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:13.912487 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:13.912565 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:13.911542 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:13.913008 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:13.915206 7009 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 LEARNER]: Raft consensus shutting down.
I20260927 15:07:13.915663 7009 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 LEARNER]: Raft consensus is shut down!
I20260927 15:07:13.915829 7009 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: stopping tablet replica
I20260927 15:07:13.933259 7009 master.cc:552] Master@127.0.97.61:34685 shutdown complete.
I20260927 15:07:13.933372 7009 master_runner.cc:305] Clearing existing system tablet
I20260927 15:07:13.933815 7009 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:13.933917 7009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:13.937660 7009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 15:07:13.939798 7119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:13.940230 7009 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:13.940706 7009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:07:13.940860 7009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
uuid: "02ff713ce03f4875a4aaf3b237c52857"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "63872ceacb893aa7fccf9d67bfd2394c"
server_key_iv: "3d709d259c627b13b14912b1e653748e"
server_key_version: "encryptionkey@0"
I20260927 15:07:13.945420 7009 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:13.952530 7009 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:07:13.952836 7009 log.cc:1196] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 15:07:13.953433 7009 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Deleting consensus metadata
I20260927 15:07:13.953751 7009 master_runner.cc:315] Copying system tablet from 127.0.97.62:34775
I20260927 15:07:13.962821 7009 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34775
I20260927 15:07:13.987886 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.964706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.97.62:34775)
0927 15:07:13.964903 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:13.964910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:13.967833 (+ 2923us) server_negotiation.cc:407] Connection header received
0927 15:07:13.968125 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:13.968131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:13.968291 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:13.968370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:13.969652 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.970768 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.972897 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.973578 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.973687 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.981044 (+ 7357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:13.981053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:13.981059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:13.981107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:13.984656 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.985551 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.985558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.985561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.985727 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:13.986203 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:13.986208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:13.986212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:13.986559 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:13.986690 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:13.987307 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 15:07:13.987534 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:07:13.989265 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:13.964689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46334)
0927 15:07:13.967685 (+ 2996us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:13.967715 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:13.967967 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:13.968584 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:13.968588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:13.969142 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:13.969468 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.969481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.971047 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.971052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.972660 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:13.972683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:13.973999 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:13.974005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:13.974195 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:13.979241 (+ 5046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:13.979278 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:13.980753 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:13.984858 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.984863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.984867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.985393 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.985882 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:13.985886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:13.985889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:13.986056 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:13.986775 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:13.986780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:13.986966 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 15:07:13.988444 (+ 1478us) client_negotiation.cc:240] Negotiation successful
0927 15:07:13.988790 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2811,"thread_start_us":2642,"threads_started":1}
I20260927 15:07:13.990355 6566 tablet_copy_service.cc:140] P 54375309f80f4087988e32e540f70f37: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 02ff713ce03f4875a4aaf3b237c52857 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46334)
I20260927 15:07:13.990612 6566 tablet_copy_service.cc:161] P 54375309f80f4087988e32e540f70f37: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 02ff713ce03f4875a4aaf3b237c52857 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46334: session id = 02ff713ce03f4875a4aaf3b237c52857-00000000000000000000000000000000
I20260927 15:07:13.995296 6566 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:07:13.997658 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:07:14.004843 7009 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: tablet copy: Starting download of 0 data blocks...
I20260927 15:07:14.005178 7009 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: tablet copy: Starting download of 1 WAL segments...
I20260927 15:07:14.008183 7009 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:07:14.013239 6566 tablet_copy_service.cc:342] P 54375309f80f4087988e32e540f70f37: Request end of tablet copy session 02ff713ce03f4875a4aaf3b237c52857-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46334
I20260927 15:07:14.013465 6566 tablet_copy_service.cc:433] P 54375309f80f4087988e32e540f70f37: ending tablet copy session 02ff713ce03f4875a4aaf3b237c52857-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 02ff713ce03f4875a4aaf3b237c52857
I20260927 15:07:14.025655 7009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:14.028415 7126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:14.028817 7127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:14.035768 7009 server_base.cc:1034] running on GCE node
W20260927 15:07:14.035835 7130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:14.036477 7009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:14.037189 7009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:14.038442 7009 hybrid_clock.cc:648] HybridClock initialized: now 1790521634038374 us; error 46 us; skew 500 ppm
I20260927 15:07:14.038678 7009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:14.041175 7009 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 15:07:14.042876 7009 webserver.cc:460] Webserver started at http://127.0.97.61:43035/ using document root <none> and password file <none>
I20260927 15:07:14.043275 7009 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:14.043360 7009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:14.046411 7009 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:14.048425 7136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:14.048877 7009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:14.049012 7009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
uuid: "02ff713ce03f4875a4aaf3b237c52857"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "63872ceacb893aa7fccf9d67bfd2394c"
server_key_iv: "3d709d259c627b13b14912b1e653748e"
server_key_version: "encryptionkey@0"
I20260927 15:07:14.049194 7009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:14.070200 7009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:14.074584 7009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:14.119879 7009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34685
I20260927 15:07:14.119901 7187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34685 every 8 connection(s)
I20260927 15:07:14.120878 7009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
I20260927 15:07:14.128274 7188 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:07:14.130960 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Bootstrap starting.
I20260927 15:07:14.142416 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:07:14.143323 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: Bootstrap complete.
I20260927 15:07:14.144181 7188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } }
I20260927 15:07:14.144404 7188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02ff713ce03f4875a4aaf3b237c52857, State: Initialized, Role: LEARNER
I20260927 15:07:14.144717 7188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } }
I20260927 15:07:14.145548 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:14.145932 7188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:14.146123 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:14.159716 7202 catalog_manager.cc:1259] Loaded cluster ID: 147140b2f02b47deaa8e84253542175e
I20260927 15:07:14.160310 7202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: loading cluster ID for follower catalog manager: success
I20260927 15:07:14.165762 7202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: acquiring CA information for follower catalog manager: success
I20260927 15:07:14.168442 7202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:07:14.188606 7003 heartbeater.cc:503] Master 127.0.97.62:34775 was elected leader, sending a full tablet report...
W20260927 15:07:14.304879 7109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:14.305177 7109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:14.305265 7109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:14.312983 7109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:14.313107 7109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:14.313153 7109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:14.313201 7109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:14.313246 7109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:14.330354 7109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:14.332522 7109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:14.334380 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:14.350041 7208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:14.349985 7207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:14.351361 7210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:14.352818 7109 server_base.cc:1034] running on GCE node
I20260927 15:07:14.354030 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:14.355916 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:14.357157 7109 hybrid_clock.cc:648] HybridClock initialized: now 1790521634357090 us; error 74 us; skew 500 ppm
I20260927 15:07:14.357530 7109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:14.361299 7109 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 15:07:14.363534 7109 webserver.cc:460] Webserver started at http://127.0.97.60:40645/ using document root <none> and password file <none>
I20260927 15:07:14.364660 7109 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:14.364817 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:14.365208 7109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:14.370404 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/instance:
uuid: "cfa1752c813343dbab208742746b3ca0"
format_stamp: "Formatted at 2026-09-27 15:07:14 on dist-test-slave-g0w1"
server_key: "1a49dbbb54f8c3eb4d51723cd92ba6e8"
server_key_iv: "4b10a97860d377d82832a2f4dbeab831"
server_key_version: "encryptionkey@0"
I20260927 15:07:14.371697 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal/instance:
uuid: "cfa1752c813343dbab208742746b3ca0"
format_stamp: "Formatted at 2026-09-27 15:07:14 on dist-test-slave-g0w1"
server_key: "1a49dbbb54f8c3eb4d51723cd92ba6e8"
server_key_iv: "4b10a97860d377d82832a2f4dbeab831"
server_key_version: "encryptionkey@0"
I20260927 15:07:14.377722 7109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:14.381434 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:14.381865 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:14.389803 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:14.392289 7109 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.004s
I20260927 15:07:14.392524 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
uuid: "cfa1752c813343dbab208742746b3ca0"
format_stamp: "Formatted at 2026-09-27 15:07:14 on dist-test-slave-g0w1"
server_key: "1a49dbbb54f8c3eb4d51723cd92ba6e8"
server_key_iv: "4b10a97860d377d82832a2f4dbeab831"
server_key_version: "encryptionkey@0"
I20260927 15:07:14.392805 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:14.396090 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.382093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.0.97.61:34685)
0927 15:07:14.388454 (+ 6361us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.388465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.389264 (+ 799us) server_negotiation.cc:407] Connection header received
0927 15:07:14.389345 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.389349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.389429 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.389568 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:14.390106 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.391607 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.393956 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.394953 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.395144 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.395352 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.395650 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.395770 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6158,"thread_start_us":5970,"threads_started":1}
I20260927 15:07:14.396108 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.381631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55530)
0927 15:07:14.388863 (+ 7232us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.388894 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.389121 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.389607 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.389612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.389640 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:14.389837 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.389846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.391806 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.391813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.393747 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.393769 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.395196 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.395205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.395403 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.395472 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.395592 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.395765 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7112,"thread_start_us":6991,"threads_started":1}
I20260927 15:07:14.397250 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54375309f80f4087988e32e540f70f37. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: true } } }
I20260927 15:07:14.397567 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:14.398128 7217 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: attempting to promote NON_VOTER 02ff713ce03f4875a4aaf3b237c52857 to VOTER
I20260927 15:07:14.399523 7217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } }
I20260927 15:07:14.401700 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 LEARNER]: Refusing update from remote peer 54375309f80f4087988e32e540f70f37: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:07:14.402340 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 15:07:14.405552 7217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 02ff713ce03f4875a4aaf3b237c52857 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } }
I20260927 15:07:14.406332 7163 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 02ff713ce03f4875a4aaf3b237c52857 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } }
I20260927 15:07:14.409804 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } }
I20260927 15:07:14.410000 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } }
I20260927 15:07:14.410151 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:14.410233 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:14.410681 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } }
I20260927 15:07:14.410894 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:14.414961 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:14.419793 7109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:14.420099 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:14.441466 7109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33959
I20260927 15:07:14.441500 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33959 every 8 connection(s)
I20260927 15:07:14.443208 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
I20260927 15:07:14.448426 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:14.452831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 7109
I20260927 15:07:14.457198 7275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:14.480221 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.461511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.0.97.62:34775)
0927 15:07:14.461714 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.461720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.462600 (+ 880us) server_negotiation.cc:407] Connection header received
0927 15:07:14.462794 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.462802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.462889 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.462962 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.465016 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.466118 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.468355 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.468815 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.468913 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.472773 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.472782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.472791 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.472841 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.477092 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.477925 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.477931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.477933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.478071 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.478492 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.478503 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.478505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.478789 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.478958 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.479622 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.479877 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:07:14.481418 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.461417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46348)
0927 15:07:14.462446 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.462500 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.462659 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.463106 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.463115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.464512 (+ 1397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.464830 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.464843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.466297 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.466302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.468198 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.468216 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.469086 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.469096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.469313 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.469963 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.470010 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.472482 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.477246 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.477260 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.477275 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.477761 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.478198 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.478204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.478208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.478360 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.478938 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.478947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.479229 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.479522 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.480227 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":157,"thread_start_us":553,"threads_started":1}
I20260927 15:07:14.481447 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.455192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.97.60:33959)
0927 15:07:14.456032 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.456049 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.456104 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:07:14.458289 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.458324 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.458933 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.459477 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.460833 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.462686 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.465157 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.466211 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.466388 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.471667 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.471677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.471691 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.471775 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.476616 (+ 4841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.477536 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.477553 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.477556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.477746 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.478178 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.478184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.478187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.478769 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.479068 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.479520 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.480060 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":595,"thread_start_us":366,"threads_started":1}
I20260927 15:07:14.483222 7275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 15:07:14.496871 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.484066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55536)
0927 15:07:14.484262 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.484284 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.484424 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.484745 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.484748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.485098 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.485248 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.485256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.486235 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.486239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.487312 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.487332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.487999 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.488004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.488191 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.488779 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.488797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.491056 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.494358 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.494362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.494365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.494830 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.495384 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.495388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.495391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.495536 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.496075 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.496079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.496215 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.496335 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.496606 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:07:14.496871 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.484149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.0.97.61:34685)
0927 15:07:14.484333 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.484341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.484363 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:07:14.484559 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.484564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.484647 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.484729 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.485369 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.486076 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.487455 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.487853 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.487946 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.491342 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.491347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.491351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.491386 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.494201 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.495012 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.495017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.495020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.495259 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.495694 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.495698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.495700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.495970 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.496094 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.496423 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.496640 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:14.498625 7275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 15:07:14.511847 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.499346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.0.1:58542)
0927 15:07:14.499588 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.499604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.499698 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.499949 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.499952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.500307 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.500454 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.500463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.501382 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.501386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.502303 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.502314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.503045 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.503050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.503301 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.503845 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.503863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.505966 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.509106 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.509114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.509120 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.509539 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.509928 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.509931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.509934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.510183 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.510912 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.510918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.511159 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.511305 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.511519 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:07:14.511932 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.499437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.0.97.60:33959)
0927 15:07:14.499585 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.499589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.499647 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:07:14.499780 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.499783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.499835 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.499911 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.500575 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.501272 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.502424 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.502927 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.503022 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.506188 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.506192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.506196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.506226 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.508918 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.509659 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.509663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.509665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.509776 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.510318 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.510325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.510330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.510772 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.510944 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.511403 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.511661 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":51}
I20260927 15:07:14.517159 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Bootstrap starting.
I20260927 15:07:14.520599 7275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:14.521777 7275 log.cc:824] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:14.525081 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: No bootstrap required, opened a new log
I20260927 15:07:14.533334 7275 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } }
I20260927 15:07:14.533775 7275 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:14.533950 7275 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa1752c813343dbab208742746b3ca0, State: Initialized, Role: FOLLOWER
I20260927 15:07:14.534619 7275 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } }
I20260927 15:07:14.535847 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } } }
I20260927 15:07:14.536327 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:14.537182 7275 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:07:14.547628 7292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:14.547787 7292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:14.552202 7109 master_runner.cc:428] Detected that this master cfa1752c813343dbab208742746b3ca0 is joining an existing cluster
I20260927 15:07:14.552376 7109 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:33959
I20260927 15:07:14.576920 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.556799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.0.97.62:34775)
0927 15:07:14.557020 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.557029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.558718 (+ 1689us) server_negotiation.cc:407] Connection header received
0927 15:07:14.559005 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.559013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.559133 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.559208 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.561231 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.562392 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.564726 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.565447 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.565558 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.568675 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.568685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.568690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.568735 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.572974 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.573891 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.573898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.573901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.574089 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.574581 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.574586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.574590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.574961 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.575134 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.576245 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.576503 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:07:14.577370 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.557868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46362)
0927 15:07:14.558609 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.558627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.558826 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.559408 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.559412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.560650 (+ 1238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.561044 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.561057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.562617 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.562623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.564539 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.564556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.565911 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.565918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.566169 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.567040 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.567129 (+ 89us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.568425 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.573191 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.573196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.573201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.573723 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.574246 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.574251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.574255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.574430 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.575253 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.575258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.575488 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.576053 (+ 565us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.576405 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":3,"thread_start_us":412,"threads_started":1}
I20260927 15:07:14.579751 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.556970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.0.97.61:34685)
0927 15:07:14.557170 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.557176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.557870 (+ 694us) server_negotiation.cc:407] Connection header received
0927 15:07:14.558064 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.558071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.558180 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.558320 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.560685 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.562311 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.565783 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.566942 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.567101 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.571210 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.571223 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.571231 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.571293 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.576063 (+ 4770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.577089 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.577097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.577101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.577275 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.577859 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.577867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.577872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.578235 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.578380 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.579004 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.579342 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:07:14.580732 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.556977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55538)
0927 15:07:14.557721 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.557746 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.557906 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.558525 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.558536 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.559409 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.560415 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.560435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.562671 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.562680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.565523 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.565547 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.567747 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.567757 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.568067 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.569113 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.569150 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.570852 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.576268 (+ 5416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.576272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.576276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.576905 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.577454 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.577459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.577462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.577686 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.578465 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.578471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.578704 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.580035 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.580381 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":31,"thread_start_us":394,"threads_started":1}
I20260927 15:07:14.588675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6607
I20260927 15:07:14.593304 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.559175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.0.1:58552)
0927 15:07:14.575535 (+ 16360us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.575555 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.576138 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.576580 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.576585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.577280 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.577635 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.577647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.579222 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.579228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.580746 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.580784 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.582070 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.582076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.582263 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.582999 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.583022 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.584223 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.590028 (+ 5805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.590033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.590038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.590646 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.591102 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.591107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.591110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.591270 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.592028 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.592037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.592222 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.592379 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.592755 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16207,"thread_start_us":16055,"threads_started":1}
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521631, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 15:07:14.599862 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.558799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.0.97.60:33959)
0927 15:07:14.559024 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.559030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.576052 (+ 17022us) server_negotiation.cc:407] Connection header received
0927 15:07:14.576245 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.576249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.576317 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.576389 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.577782 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.579017 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.580920 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.581692 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.581810 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.585476 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.585483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.585489 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.585526 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.589874 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.590780 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.590787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.590790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.590954 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.591403 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.591407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.591411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.591804 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.591973 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.598289 (+ 6316us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.599296 (+ 1007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:07:14.607316 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.584890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.0.97.60:33959)
0927 15:07:14.585569 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.585577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.585614 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:14.585712 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.585717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.585813 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.585943 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.586976 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.588669 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.590810 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.591645 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.596825 (+ 5180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.597167 (+ 342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.597173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.597179 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.597215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.603487 (+ 6272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.604561 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.604568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.604571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.604788 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.605319 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.605325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.605328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.605765 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.605922 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.606494 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.606814 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":367,"threads_started":1}
I20260927 15:07:14.607913 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.584890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.0.1:58564)
0927 15:07:14.585096 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.585118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.585247 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.586004 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.586010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.586575 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.586777 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.586789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.588979 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.588985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.590605 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.590622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.591825 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.591835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.592050 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.592776 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.592805 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.596593 (+ 3788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.603714 (+ 7121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.603720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.603725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.604375 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.604972 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.604980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.604984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.605181 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.605945 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.605950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.606185 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.606351 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.606708 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:07:14.609932 6546 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:33959: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cfa1752c813343dbab208742746b3ca0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 33959
}
attrs {
promote: true
}
}
dest_uuid: "54375309f80f4087988e32e540f70f37"
cas_config_opid_index: 6
I20260927 15:07:14.610810 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } }
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 15:07:14.613555 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Refusing update from remote peer 54375309f80f4087988e32e540f70f37: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:07:14.613910 7250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:14.614264 7218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 15:07:14.618134 7217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cfa1752c813343dbab208742746b3ca0 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:14.618783 7163 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cfa1752c813343dbab208742746b3ca0 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:14.622431 7217 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:33959
I20260927 15:07:14.622797 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:14.623162 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:14.623420 7109 master.cc:537] Master@127.0.97.60:33959 shutting down...
I20260927 15:07:14.623944 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.590420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.0.97.1:41133)
0927 15:07:14.600599 (+ 10179us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.600611 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.600649 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:07:14.600730 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.600738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.600989 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.601245 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.602887 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.604811 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.606769 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.608326 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.609130 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.614647 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.614667 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.614689 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.614744 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.619097 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.620138 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.620150 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.620154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.620330 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.620822 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.620827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.620830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.621289 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.622699 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.622877 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.623267 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9967,"thread_start_us":9785,"threads_started":1}
I20260927 15:07:14.624118 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:14.624399 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:14.626246 6730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:14.627869 7250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Refusing update from remote peer 54375309f80f4087988e32e540f70f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:07:14.628902 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:14.632593 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:14.632936 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
W20260927 15:07:14.655956 6533 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 15:07:14.658566 6533 consensus_peers.cc:487] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 -> Peer cfa1752c813343dbab208742746b3ca0 (127.0.97.60:33959): Couldn't send request to peer cfa1752c813343dbab208742746b3ca0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:07:14.660558 7109 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:14.660996 7109 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:14.661226 7109 tablet_replica.cc:330] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: stopping tablet replica
I20260927 15:07:14.667704 7109 master.cc:552] Master@127.0.97.60:33959 shutdown complete.
I20260927 15:07:14.667809 7109 master_runner.cc:305] Clearing existing system tablet
I20260927 15:07:14.668174 7109 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:14.668282 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:14.670512 7109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 15:07:14.672040 7306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:14.672363 7109 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:14.672652 7109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:14.672747 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
uuid: "cfa1752c813343dbab208742746b3ca0"
format_stamp: "Formatted at 2026-09-27 15:07:14 on dist-test-slave-g0w1"
server_key: "1a49dbbb54f8c3eb4d51723cd92ba6e8"
server_key_iv: "4b10a97860d377d82832a2f4dbeab831"
server_key_version: "encryptionkey@0"
I20260927 15:07:14.676168 7109 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:14.680786 7109 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:07:14.680962 7109 log.cc:1196] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 15:07:14.681456 7109 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Deleting consensus metadata
I20260927 15:07:14.681708 7109 master_runner.cc:315] Copying system tablet from 127.0.97.62:34775
I20260927 15:07:14.684450 7109 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34775
I20260927 15:07:14.697741 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.685262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.0.97.62:34775)
0927 15:07:14.685530 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:14.685540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:14.686175 (+ 635us) server_negotiation.cc:407] Connection header received
0927 15:07:14.686398 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:14.686404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:14.686492 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:14.686606 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:14.687553 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.688275 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.689561 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.689979 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.690130 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.692077 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:14.692090 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:14.692094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:14.692126 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:14.694986 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.695724 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.695732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.695735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.695873 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:14.696269 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:14.696275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:14.696278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:14.696551 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:14.696714 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:14.697133 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 15:07:14.697373 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:07:14.697825 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:14.685248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46378)
0927 15:07:14.685975 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:14.686001 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:14.686260 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:14.686628 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:14.686632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:14.687187 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:14.687401 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.687410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.688411 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.688419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.689409 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:14.689421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:14.690160 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:14.690166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:14.690361 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:14.691063 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:14.691129 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:14.691858 (+ 729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:14.695189 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.695195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.695199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.695593 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.695991 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:14.695996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:14.696000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:14.696154 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:14.696675 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:14.696681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:14.696886 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 15:07:14.697032 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 15:07:14.697384 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":455,"threads_started":1}
I20260927 15:07:14.698697 6566 tablet_copy_service.cc:140] P 54375309f80f4087988e32e540f70f37: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cfa1752c813343dbab208742746b3ca0 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46378)
I20260927 15:07:14.698874 6566 tablet_copy_service.cc:161] P 54375309f80f4087988e32e540f70f37: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cfa1752c813343dbab208742746b3ca0 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46378: session id = cfa1752c813343dbab208742746b3ca0-00000000000000000000000000000000
I20260927 15:07:14.701171 6566 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:07:14.702801 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:07:14.707828 7109 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: tablet copy: Starting download of 0 data blocks...
I20260927 15:07:14.708175 7109 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: tablet copy: Starting download of 1 WAL segments...
I20260927 15:07:14.710508 7109 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:07:14.714345 6566 tablet_copy_service.cc:342] P 54375309f80f4087988e32e540f70f37: Request end of tablet copy session cfa1752c813343dbab208742746b3ca0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46378
I20260927 15:07:14.714506 6566 tablet_copy_service.cc:433] P 54375309f80f4087988e32e540f70f37: ending tablet copy session cfa1752c813343dbab208742746b3ca0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cfa1752c813343dbab208742746b3ca0
I20260927 15:07:14.725234 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:14.727977 7313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:14.738945 7316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:14.739419 7109 server_base.cc:1034] running on GCE node
W20260927 15:07:14.739563 7314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:14.740026 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:14.740692 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:14.742018 7109 hybrid_clock.cc:648] HybridClock initialized: now 1790521634741978 us; error 31 us; skew 500 ppm
I20260927 15:07:14.742312 7109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:14 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:14.744423 7109 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 15:07:14.752604 7109 webserver.cc:460] Webserver started at http://127.0.97.60:33125/ using document root <none> and password file <none>
I20260927 15:07:14.752965 7109 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:14.753146 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:14.755573 7109 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:07:14.757243 7323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:14.757686 7109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260927 15:07:14.757818 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
uuid: "cfa1752c813343dbab208742746b3ca0"
format_stamp: "Formatted at 2026-09-27 15:07:14 on dist-test-slave-g0w1"
server_key: "1a49dbbb54f8c3eb4d51723cd92ba6e8"
server_key_iv: "4b10a97860d377d82832a2f4dbeab831"
server_key_version: "encryptionkey@0"
I20260927 15:07:14.757999 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:14.769099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6607
I20260927 15:07:14.771992 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:14.776098 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:14.794665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41133
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34775
--webserver_port=42721
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:14.798689 7109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33959
I20260927 15:07:14.798852 7374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33959 every 8 connection(s)
I20260927 15:07:14.799659 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
I20260927 15:07:14.806627 7375 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:07:14.811784 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Bootstrap starting.
I20260927 15:07:14.828111 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:07:14.828733 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Bootstrap complete.
I20260927 15:07:14.829707 7375 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } }
I20260927 15:07:14.829959 7375 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfa1752c813343dbab208742746b3ca0, State: Initialized, Role: LEARNER
I20260927 15:07:14.830197 7375 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } }
I20260927 15:07:14.833917 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:14.834267 7375 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:14.834502 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:14.861752 7390 catalog_manager.cc:1259] Loaded cluster ID: 147140b2f02b47deaa8e84253542175e
I20260927 15:07:14.861876 7390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: loading cluster ID for follower catalog manager: success
I20260927 15:07:14.864763 7390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: acquiring CA information for follower catalog manager: success
I20260927 15:07:14.866848 7390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:07:15.172806 7377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:15.173121 7377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:15.173291 7377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:15.180641 7377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:15.180797 7377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:15.180971 7377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:15.191856 7377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42721
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:15.193375 7377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:15.194620 7377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:15.208392 7396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:15.208420 7398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:15.208634 7395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:15.210049 7377 server_base.cc:1034] running on GCE node
I20260927 15:07:15.210667 7377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:15.211714 7377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:15.212946 7377 hybrid_clock.cc:648] HybridClock initialized: now 1790521635212902 us; error 59 us; skew 500 ppm
I20260927 15:07:15.213271 7377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:15.216478 7377 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 15:07:15.217864 7377 webserver.cc:460] Webserver started at http://127.0.97.1:42721/ using document root <none> and password file <none>
I20260927 15:07:15.218569 7377 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:15.218732 7377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:15.225531 7377 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:15.228672 7408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:15.229995 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.222659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.0.1:58566)
0927 15:07:15.223185 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.223213 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.223383 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.223811 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.223815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.223836 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:15.224002 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.224009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.226237 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.226242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.227679 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.227691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.229351 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.229358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.229570 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.229596 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.229682 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.229779 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":207,"threads_started":1}
I20260927 15:07:15.230194 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.222919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.0.97.60:33959)
0927 15:07:15.223460 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.223468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.223491 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:07:15.223566 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.223571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.223659 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.223788 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:15.224177 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.226098 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.227818 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.229116 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.229249 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.229484 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.229726 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.229852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260927 15:07:15.230368 7377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:15.231230 7350 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260927 15:07:15.230526 7377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "d11403f429944d3d850732cf48ea6fff"
format_stamp: "Formatted at 2026-09-27 15:07:11 on dist-test-slave-g0w1"
server_key: "35ca544a095095b446127263595df7d0"
server_key_iv: "f4ffb76d03b3a8cf0c2d389da7e16b43"
server_key_version: "encryptionkey@0"
I20260927 15:07:15.231725 7377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:15.231740 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54375309f80f4087988e32e540f70f37. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:15.231990 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:15.247581 7377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:15.252808 7377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:15.253136 7377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:15.254127 7377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:15.255503 7377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:15.255597 7377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:15.255721 7377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:15.255806 7377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:15.292971 7377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41133
I20260927 15:07:15.293033 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41133 every 8 connection(s)
I20260927 15:07:15.294523 7377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:15.302347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 7377
I20260927 15:07:15.302668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6741
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:15.325160 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.300668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35583 (local address 127.0.97.60:33959)
0927 15:07:15.300841 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.300847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.301100 (+ 253us) server_negotiation.cc:407] Connection header received
0927 15:07:15.302443 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.302449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.302543 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.302614 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.305529 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.306651 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.309740 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.310382 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.310485 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.316697 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.316704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.316709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.316752 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.320828 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.322075 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.322081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.322085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.322228 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.322970 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.322975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.322978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.323375 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.323513 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.324591 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.324820 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:07:15.326526 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.300399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.97.1:35583)
0927 15:07:15.300952 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.300971 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.302127 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.303091 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.303105 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.304222 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:15.305308 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.305335 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.306840 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.306846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.309555 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.309570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.311322 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.311329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.311762 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.312777 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:15.312819 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:15.316418 (+ 3599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:15.321017 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.321020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.321028 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.321922 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.322346 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.322349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.322351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.322832 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.323603 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:15.323611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:15.324091 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.325413 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.325700 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":441,"spinlock_wait_cycles":25088,"thread_start_us":267,"threads_started":1}
I20260927 15:07:15.327271 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.297882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46249 (local address 127.0.97.61:34685)
0927 15:07:15.299068 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.299074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.299891 (+ 817us) server_negotiation.cc:407] Connection header received
0927 15:07:15.302530 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.302536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.302622 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.302715 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.305517 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.306651 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.309732 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.310382 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.310490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.316701 (+ 6211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.316707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.316711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.316752 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.320842 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.322075 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.322081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.322085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.322696 (+ 611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.323282 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.323287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.323290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.323608 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.323771 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.326601 (+ 2830us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.326842 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":950,"threads_started":1}
I20260927 15:07:15.327582 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.298639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.97.1:46249)
0927 15:07:15.299691 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.299765 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.302127 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.303091 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.303105 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.304272 (+ 1167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:15.305308 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.305335 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.306840 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.306846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.309555 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.309570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.311322 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.311329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.311760 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.312839 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:15.312856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:15.316418 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:15.321000 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.321009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.321028 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.321922 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.322867 (+ 945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.322874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.322881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.323144 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.323824 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:15.323828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:15.324091 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.324442 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.325019 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":384,"spinlock_wait_cycles":29056,"thread_start_us":464,"threads_started":1}
I20260927 15:07:15.328353 7521 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33959
I20260927 15:07:15.328686 7521 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:15.328851 7523 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34685
I20260927 15:07:15.329025 7523 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:15.329835 7523 heartbeater.cc:511] Master 127.0.97.61:34685 requested a full tablet report, sending...
I20260927 15:07:15.330833 7521 heartbeater.cc:511] Master 127.0.97.60:33959 requested a full tablet report, sending...
I20260927 15:07:15.332911 7153 ts_manager.cc:194] Registered new tserver with Master: d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133)
I20260927 15:07:15.333009 7340 ts_manager.cc:194] Registered new tserver with Master: d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133)
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 15:07:15.334332 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.297595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33205 (local address 127.0.97.62:34775)
0927 15:07:15.298348 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.298355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.300302 (+ 1947us) server_negotiation.cc:407] Connection header received
0927 15:07:15.302643 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.302649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.302737 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.302842 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.313270 (+ 10428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.314885 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.319449 (+ 4564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.320111 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.320246 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.325234 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.325242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.325246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.325290 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.330429 (+ 5139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.331582 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.331589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.331593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.331767 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.332246 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.332251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.332254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.332638 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.332776 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.333569 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.333905 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":493,"threads_started":1}
I20260927 15:07:15.335258 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.298644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:33205)
0927 15:07:15.299691 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.299765 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.302115 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.303148 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.303156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.311410 (+ 8254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:15.312667 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.312685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.315370 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.315379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.318068 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.318088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.320505 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.320512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.320738 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.321386 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:15.321410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:15.324958 (+ 3548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:15.330748 (+ 5790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.330758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.330762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.331422 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.331930 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.331935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.331939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.332108 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.332870 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:15.332875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:15.333147 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.334529 (+ 1382us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.334864 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":860,"spinlock_wait_cycles":55680,"thread_start_us":464,"threads_started":1}
I20260927 15:07:15.336225 7522 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34775
I20260927 15:07:15.336352 7522 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:15.336777 7522 heartbeater.cc:511] Master 127.0.97.62:34775 requested a full tablet report, sending...
I20260927 15:07:15.337878 6546 ts_manager.cc:194] Re-registered known tserver with Master: d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133)
I20260927 15:07:15.338683 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33205
I20260927 15:07:15.341959 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.303578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.0.97.2:42201)
0927 15:07:15.319886 (+ 16308us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.319896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.319928 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:07:15.319998 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.320005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.320331 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.323019 (+ 2688us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.323300 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.324891 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.326954 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.327877 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.328011 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.334638 (+ 6627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.334652 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.334666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.334704 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.338654 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.339660 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.339674 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.339680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.339863 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.340322 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.340327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.340331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.340767 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.341076 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.341323 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.341587 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16123,"thread_start_us":18055,"threads_started":1}
I20260927 15:07:15.343343 6864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:15.431650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6741
I20260927 15:07:15.448547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42201
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34775
--webserver_port=45185
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:15.789595 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:15.789896 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:15.790063 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:15.797917 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:15.798089 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:15.798281 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:15.804845 7404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: true } } }
I20260927 15:07:15.807330 7404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:15.814141 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45185
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:15.814334 7538 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: attempting to promote NON_VOTER cfa1752c813343dbab208742746b3ca0 to VOTER
I20260927 15:07:15.815145 7538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } }
I20260927 15:07:15.816005 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:15.817847 7350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 LEARNER]: Refusing update from remote peer 54375309f80f4087988e32e540f70f37: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 15:07:15.817899 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:15.817937 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Refusing update from remote peer 54375309f80f4087988e32e540f70f37: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 15:07:15.818600 7537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:07:15.819118 7404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:07:15.822273 7537 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cfa1752c813343dbab208742746b3ca0 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.823098 7163 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cfa1752c813343dbab208742746b3ca0 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.825326 7404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.825287 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.825567 7404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:15.825585 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:15.826663 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.826897 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:15.827953 7350 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cfa1752c813343dbab208742746b3ca0 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.834270 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.835070 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:15.836741 7545 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54375309f80f4087988e32e540f70f37" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:15.837113 7545 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 15:07:15.843806 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:15.843909 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:15.843895 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:15.844973 7533 server_base.cc:1034] running on GCE node
I20260927 15:07:15.845789 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:15.846969 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:15.848245 7533 hybrid_clock.cc:648] HybridClock initialized: now 1790521635848172 us; error 68 us; skew 500 ppm
I20260927 15:07:15.848706 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:15.852913 7533 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 15:07:15.854652 7533 webserver.cc:460] Webserver started at http://127.0.97.2:45185/ using document root <none> and password file <none>
I20260927 15:07:15.855499 7533 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:15.855677 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:15.862231 7533 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:15.865566 7556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:15.867328 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:15.867512 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "dc9c744a346d4c3fa599ef02298835ae"
format_stamp: "Formatted at 2026-09-27 15:07:12 on dist-test-slave-g0w1"
server_key: "96e0ec78cc473fa3d78ff128980d29c3"
server_key_iv: "b38045bb4c9eaf941c0796800438269e"
server_key_version: "encryptionkey@0"
I20260927 15:07:15.868515 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:15.887656 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:15.892019 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:15.892323 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:15.893375 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:15.894812 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:15.894901 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:15.895056 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:15.895205 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:07:15.928699 7533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42201
I20260927 15:07:15.928800 7668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42201 every 8 connection(s)
I20260927 15:07:15.930262 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:15.931269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 7533
I20260927 15:07:15.931558 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6875
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 15:07:15.965832 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.933228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.97.3:43795)
0927 15:07:15.933941 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.933950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.933982 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:07:15.934057 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.934066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.934347 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.937208 (+ 2861us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.937531 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.939653 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.943167 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.944603 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.947978 (+ 3375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.956123 (+ 8145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.956149 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.956177 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.956243 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.960941 (+ 4698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.961856 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.961869 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.961874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.962155 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.962743 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.962749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.962752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.963294 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.964791 (+ 1497us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.965028 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.965337 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":518,"threads_started":1}
I20260927 15:07:15.967955 6998 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 15:07:15 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521635, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 15:07:15.986011 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.935034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44133 (local address 127.0.97.62:34775)
0927 15:07:15.951482 (+ 16448us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.951489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.951518 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:07:15.951571 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.951575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.951662 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.951806 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.954550 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.956215 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.969544 (+ 13329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.970477 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.970640 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.976312 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.976319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.976324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.976387 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.980999 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.982645 (+ 1646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.982651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.982654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.982855 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.983746 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.983755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.983761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.984152 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.984310 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.985364 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.985627 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16316,"thread_start_us":16172,"threads_started":1}
I20260927 15:07:15.986299 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.935454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52131 (local address 127.0.97.61:34685)
0927 15:07:15.940338 (+ 4884us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.940347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.952473 (+ 12126us) server_negotiation.cc:407] Connection header received
0927 15:07:15.952658 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.952664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.952764 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.952897 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.955395 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.963708 (+ 8313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.972207 (+ 8499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.972792 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.972971 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.977946 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.977955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.977962 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.978008 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.981551 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.982844 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.982849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.982852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.983008 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.984335 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.984341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.984344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.984745 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.984906 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.985683 (+ 777us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.985912 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4667,"thread_start_us":4526,"threads_started":1}
I20260927 15:07:15.987731 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.934310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:44133)
0927 15:07:15.945620 (+ 11310us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.945709 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.950004 (+ 4295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.952203 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.952219 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.953215 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:15.954338 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.954382 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.964043 (+ 9661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.964049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.969303 (+ 5254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.969326 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.971083 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.971094 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.971505 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.972531 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:15.972565 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:15.976027 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:15.981743 (+ 5716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.981748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.981751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.982502 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.983004 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.983011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.983016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.983410 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.984339 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:15.984353 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:15.984915 (+ 562us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.985261 (+ 346us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.985924 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10822,"mutex_wait_us":52,"spinlock_wait_cycles":29824,"thread_start_us":17587,"threads_started":1}
I20260927 15:07:15.987733 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.935132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.97.2:52131)
0927 15:07:15.952360 (+ 17228us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.952380 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.952522 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.952993 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.952998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.953535 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:15.955199 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.955214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.965893 (+ 10679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.966014 (+ 121us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.969113 (+ 3099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.969136 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.973021 (+ 3885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.973028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.973206 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.973890 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:15.973918 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:15.977696 (+ 3778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:15.981726 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.981734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.981740 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.982674 (+ 934us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.983165 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.983171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.983175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.984168 (+ 993us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.984972 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:15.984976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:15.985124 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.985283 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.986783 (+ 1500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17087,"mutex_wait_us":1930,"spinlock_wait_cycles":1792,"thread_start_us":9818,"threads_started":1}
I20260927 15:07:15.989537 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.952728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36191 (local address 127.0.97.60:33959)
0927 15:07:15.959625 (+ 6897us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:15.959644 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:15.962590 (+ 2946us) server_negotiation.cc:407] Connection header received
0927 15:07:15.962680 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:15.962687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:15.962775 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:15.962884 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:15.964435 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.965684 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.969660 (+ 3976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.971033 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.971230 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.977152 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:15.977162 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:15.977168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:15.977220 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:15.981274 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.983166 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.983171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.983175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.984473 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:15.984953 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:15.984959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:15.984962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:15.986882 (+ 1920us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:15.987032 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:15.989043 (+ 2011us) server_negotiation.cc:299] Negotiation successful
0927 15:07:15.989239 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6735,"thread_start_us":6576,"threads_started":1}
I20260927 15:07:15.989584 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34775
I20260927 15:07:15.989939 7670 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:15.990358 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:15.954047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.97.2:36191)
0927 15:07:15.962296 (+ 8249us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:15.962319 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:15.962461 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:15.963145 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:15.963150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:15.963796 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:15.964218 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.964237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.966323 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.966329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.969448 (+ 3119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:15.969465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:15.971549 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:15.971559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:15.971852 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:15.973421 (+ 1569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:15.973448 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:15.976853 (+ 3405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:15.981478 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.981489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.981510 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.982502 (+ 992us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.984616 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:15.984621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:15.984625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:15.984814 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:15.987137 (+ 2323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:15.987142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:15.987289 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 15:07:15.987456 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 15:07:15.987770 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8091,"mutex_wait_us":769,"spinlock_wait_cycles":6016,"thread_start_us":7925,"threads_started":1}
I20260927 15:07:15.990988 7670 heartbeater.cc:511] Master 127.0.97.62:34775 requested a full tablet report, sending...
I20260927 15:07:15.991166 7671 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34685
I20260927 15:07:15.991297 7671 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:15.991755 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33959
I20260927 15:07:15.991806 7671 heartbeater.cc:511] Master 127.0.97.61:34685 requested a full tablet report, sending...
I20260927 15:07:15.991894 7669 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:15.992313 7669 heartbeater.cc:511] Master 127.0.97.60:33959 requested a full tablet report, sending...
I20260927 15:07:15.993688 6546 ts_manager.cc:194] Re-registered known tserver with Master: dc9c744a346d4c3fa599ef02298835ae (127.0.97.2:42201)
I20260927 15:07:15.993829 7340 ts_manager.cc:194] Registered new tserver with Master: dc9c744a346d4c3fa599ef02298835ae (127.0.97.2:42201)
I20260927 15:07:15.993997 7153 ts_manager.cc:194] Registered new tserver with Master: dc9c744a346d4c3fa599ef02298835ae (127.0.97.2:42201)
I20260927 15:07:15.994839 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44133
I20260927 15:07:16.062058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6875
I20260927 15:07:16.078548 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43795
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34775
--webserver_port=46191
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:16.341921 7522 heartbeater.cc:503] Master 127.0.97.62:34775 was elected leader, sending a full tablet report...
W20260927 15:07:16.430456 7682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:16.430763 7682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:16.430859 7682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:16.438563 7682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:16.438706 7682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:16.438822 7682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:16.449209 7682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46191
--tserver_master_addrs=127.0.97.62:34775,127.0.97.61:34685,127.0.97.60:33959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:16.450697 7682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:16.452158 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:16.467154 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:16.467605 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:16.467664 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:16.468560 7682 server_base.cc:1034] running on GCE node
I20260927 15:07:16.469333 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:16.470350 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:16.471604 7682 hybrid_clock.cc:648] HybridClock initialized: now 1790521636471548 us; error 53 us; skew 500 ppm
I20260927 15:07:16.471952 7682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:16 dist-test-slave-g0w1 krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:16.475266 7682 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 15:07:16.476818 7682 webserver.cc:460] Webserver started at http://127.0.97.3:46191/ using document root <none> and password file <none>
I20260927 15:07:16.477540 7682 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:16.477712 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:16.483393 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:16.486277 7698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:16.487911 7682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:16.488096 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "2a8748a6950c4056bd5e326e4c250701"
format_stamp: "Formatted at 2026-09-27 15:07:13 on dist-test-slave-g0w1"
server_key: "e6eea981564d90585d83efbbf05006a0"
server_key_iv: "cb94797692b4ab4d765531363ed11f84"
server_key_version: "encryptionkey@0"
I20260927 15:07:16.489178 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:16.505684 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:16.510309 7682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:16.510562 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:16.511572 7682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:16.512916 7682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:16.513005 7682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:16.513082 7682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:16.513146 7682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:16.546190 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43795
I20260927 15:07:16.546247 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43795 every 8 connection(s)
I20260927 15:07:16.547912 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:16.550858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 7682
Sep 27 15:07:16 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 15:07:16 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 15:07:16.593847 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.552657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43481 (local address 127.0.97.62:34775)
0927 15:07:16.564580 (+ 11923us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.564590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.564621 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:07:16.566295 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.566303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.566410 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.566521 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:16.569951 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.571178 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.575292 (+ 4114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.576008 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.576143 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.584211 (+ 8068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:16.584219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:16.584226 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:16.584285 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:16.588886 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.591510 (+ 2624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.591514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.591517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.591690 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.592165 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.592170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.592173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.592526 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:16.592674 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.593231 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.593466 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11751,"thread_start_us":11625,"threads_started":1}
I20260927 15:07:16.594300 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.566178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34013 (local address 127.0.97.60:33959)
0927 15:07:16.569544 (+ 3366us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.569549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.569563 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:16.569612 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.569616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.569681 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.569811 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:16.570869 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.575661 (+ 4792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.579024 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.579885 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.580006 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.584633 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:16.584643 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:16.584650 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:16.584699 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:16.588886 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.589981 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.589987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.589991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.590158 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.590729 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.590734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.590737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.591111 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:16.591273 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.593253 (+ 1980us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.593581 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3273,"thread_start_us":3179,"threads_started":1}
I20260927 15:07:16.595881 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.551847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:43481)
0927 15:07:16.561736 (+ 9889us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.561799 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.565905 (+ 4106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.567401 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.567420 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.568706 (+ 1286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:16.569752 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.569774 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.571997 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.572004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.575122 (+ 3118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.575142 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.576382 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.576393 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.576908 (+ 515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.578046 (+ 1138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:16.578092 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:16.583915 (+ 5823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:16.590704 (+ 6789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:16.590712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:16.590719 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:16.591379 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:16.591806 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:16.591813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:16.591820 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:16.592031 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:16.592661 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:16.592670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:16.592919 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.593136 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.593915 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9530,"mutex_wait_us":4,"spinlock_wait_cycles":27776,"thread_start_us":13472,"threads_started":1}
Sep 27 15:07:16 dist-test-slave-g0w1 krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 15:07:16.597025 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.566148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.97.3:34013)
0927 15:07:16.569071 (+ 2923us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.569089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.569227 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.569835 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.569839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.570448 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:16.570714 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.570723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.576696 (+ 5973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.576704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.578806 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.578826 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.580350 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.580358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.580573 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.581822 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:16.581846 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:16.584364 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:16.589106 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:16.589118 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:16.589141 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:16.589832 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:16.590293 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:16.590301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:16.590308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:16.590575 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:16.591363 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:16.591371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:16.591899 (+ 528us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.594834 (+ 2935us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.595164 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2770,"spinlock_wait_cycles":5376,"thread_start_us":2608,"threads_started":1}
I20260927 15:07:16.597992 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34775
I20260927 15:07:16.598352 7812 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:16.599794 7812 heartbeater.cc:511] Master 127.0.97.62:34775 requested a full tablet report, sending...
I20260927 15:07:16.602301 6546 ts_manager.cc:194] Re-registered known tserver with Master: 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795)
I20260927 15:07:16.603454 6546 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43481
I20260927 15:07:16.603832 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33959
I20260927 15:07:16.603962 7811 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:16.604434 7811 heartbeater.cc:511] Master 127.0.97.60:33959 requested a full tablet report, sending...
I20260927 15:07:16.606315 7340 ts_manager.cc:194] Registered new tserver with Master: 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795)
I20260927 15:07:16.611511 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.552198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40405 (local address 127.0.97.61:34685)
0927 15:07:16.563536 (+ 11338us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.563547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.565897 (+ 2350us) server_negotiation.cc:407] Connection header received
0927 15:07:16.567274 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.567283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.567400 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.567507 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:16.572026 (+ 4519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.573081 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.575543 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.576346 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.576466 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.601721 (+ 25255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:16.601731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:16.601738 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:16.601806 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:16.605579 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.606432 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.606437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.606442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.608751 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.609261 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.609266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.609273 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.609673 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:16.610658 (+ 985us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.610803 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.611051 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11164,"thread_start_us":11061,"threads_started":1}
I20260927 15:07:16.611686 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.551847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.97.3:40405)
0927 15:07:16.565763 (+ 13916us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.565782 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.565973 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.567868 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.567876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.571508 (+ 3632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:16.571807 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.571823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.573334 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.573343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.575322 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.575340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.581194 (+ 5854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.581203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.581438 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.583086 (+ 1648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:16.583111 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:16.599135 (+ 16024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:16.605768 (+ 6633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:16.605776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:16.605783 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:16.606286 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:16.608900 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:16.608905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:16.608909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:16.609119 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:16.609867 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:16.609871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:16.610033 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.610160 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.611239 (+ 1079us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13763,"mutex_wait_us":388,"thread_start_us":9026,"threads_started":1}
I20260927 15:07:16.612660 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34685
I20260927 15:07:16.612829 7813 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:16.613324 7813 heartbeater.cc:511] Master 127.0.97.61:34685 requested a full tablet report, sending...
I20260927 15:07:16.613966 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.572707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.0.97.62:34775)
0927 15:07:16.580793 (+ 8086us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.580801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.582176 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 15:07:16.582259 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.582268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.582373 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.582507 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:16.584982 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.590204 (+ 5222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.594989 (+ 4785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.595650 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.595747 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.600964 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:16.600972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:16.600979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:16.601029 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:16.608538 (+ 7509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.610244 (+ 1706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.610249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.610252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.610439 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.611938 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.611946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.611952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.612331 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:16.612585 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.613248 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.613567 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7949,"thread_start_us":7818,"threads_started":1}
I20260927 15:07:16.614585 7153 ts_manager.cc:194] Registered new tserver with Master: 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795)
I20260927 15:07:16.615737 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.572989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.0.97.61:34685)
0927 15:07:16.576175 (+ 3186us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.576181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.584735 (+ 8554us) server_negotiation.cc:407] Connection header received
0927 15:07:16.585405 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.585410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.585499 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.585590 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:16.588798 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.590505 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.599444 (+ 8939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.600450 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.600592 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.605933 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:16.605940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:16.605944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:16.605982 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:16.610506 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.611424 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.611429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.611432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.611809 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.612294 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.612299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.612302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.613738 (+ 1436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:16.613989 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.615157 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.615395 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3047,"thread_start_us":2905,"threads_started":1}
I20260927 15:07:16.626349 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.585685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.0.97.60:33959)
0927 15:07:16.594792 (+ 9107us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.594797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.610883 (+ 16086us) server_negotiation.cc:407] Connection header received
0927 15:07:16.611188 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.611194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.611287 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.611356 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:16.613880 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.614915 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.616788 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.617396 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.617478 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.620737 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:16.620742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:16.620746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:16.620795 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:16.623571 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.624299 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.624304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.624307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.624442 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:16.624866 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:16.624870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:16.624872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:16.625154 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:16.625417 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.625757 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.625999 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8995,"thread_start_us":6268,"threads_started":1}
I20260927 15:07:16.619899 6546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:16.628466 6546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:16.692269 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.649075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.0.97.3:43795)
0927 15:07:16.678008 (+ 28933us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.678023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.679185 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 15:07:16.679628 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.679642 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.680029 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.680308 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:16.681584 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.683865 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.688796 (+ 4931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.691050 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.691234 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.691471 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.691735 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.691923 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28651,"thread_start_us":28999,"threads_started":1}
I20260927 15:07:16.692251 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.648537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43795 (local address 127.0.0.1:51570)
0927 15:07:16.678942 (+ 30405us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.678978 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.679361 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.680696 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.680704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.680748 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:16.681288 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.681304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.686389 (+ 5085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.686402 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.688570 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.688589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.691301 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.691308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.691509 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.691546 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.691652 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.691796 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30213,"mutex_wait_us":10,"thread_start_us":37590,"threads_started":1}
I20260927 15:07:16.694880 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.648838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42201 (local address 127.0.0.1:57488)
0927 15:07:16.684642 (+ 35804us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.684671 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.684829 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.686057 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.686066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.686109 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:16.686723 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.686741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.689400 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.689406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.691567 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.691579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.693324 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.693330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.694332 (+ 1002us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.694364 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.694467 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.694603 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35581,"mutex_wait_us":29,"thread_start_us":35395,"threads_started":1}
I20260927 15:07:16.695050 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.649448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.97.2:42201)
0927 15:07:16.685333 (+ 35885us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.685342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.685372 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:07:16.685466 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.685472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.685795 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.686005 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:16.686981 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.689242 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.691780 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.693098 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.693264 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.693890 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.694594 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.694793 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35734,"thread_start_us":35614,"threads_started":1}
I20260927 15:07:16.697885 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.648359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.0.97.1:41133)
0927 15:07:16.689172 (+ 40813us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.689185 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.689217 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:07:16.689323 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.689335 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.689662 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.689902 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:16.690525 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.691809 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.695755 (+ 3946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.696830 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.696999 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.697218 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.697425 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.697584 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40618,"thread_start_us":40467,"threads_started":1}
I20260927 15:07:16.697927 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.648031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41133 (local address 127.0.0.1:35998)
0927 15:07:16.648750 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.648776 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.648909 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.689884 (+ 40975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.689888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.689915 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:16.690359 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.690368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.691959 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.691964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.694180 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.694195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.697036 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.697042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.697218 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.697259 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.697383 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.697522 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":1,"thread_start_us":421,"threads_started":1}
I20260927 15:07:16.698820 7604 tablet_service.cc:1467] Processing CreateTablet for tablet 8346f23a6741487eac2c02051e79d2a7 (DEFAULT_TABLE table=test-table [id=29ec6cf6cc1b4458a67023799edf0ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:16.698822 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 8346f23a6741487eac2c02051e79d2a7 (DEFAULT_TABLE table=test-table [id=29ec6cf6cc1b4458a67023799edf0ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:16.700418 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8346f23a6741487eac2c02051e79d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:16.700420 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8346f23a6741487eac2c02051e79d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:16.710994 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 8346f23a6741487eac2c02051e79d2a7 (DEFAULT_TABLE table=test-table [id=29ec6cf6cc1b4458a67023799edf0ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:16.712394 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8346f23a6741487eac2c02051e79d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:16.718235 7845 tablet_bootstrap.cc:492] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Bootstrap starting.
I20260927 15:07:16.720708 7846 tablet_bootstrap.cc:492] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Bootstrap starting.
I20260927 15:07:16.723688 7845 tablet_bootstrap.cc:654] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:16.724403 7846 tablet_bootstrap.cc:654] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:16.725047 7845 log.cc:824] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:16.725422 7846 log.cc:824] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:16.728248 7846 tablet_bootstrap.cc:492] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: No bootstrap required, opened a new log
I20260927 15:07:16.728574 7846 ts_tablet_manager.cc:1397] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:07:16.730381 7849 tablet_bootstrap.cc:492] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Bootstrap starting.
I20260927 15:07:16.731837 7845 tablet_bootstrap.cc:492] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: No bootstrap required, opened a new log
I20260927 15:07:16.732566 7845 ts_tablet_manager.cc:1397] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260927 15:07:16.734764 7849 tablet_bootstrap.cc:654] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:16.736159 7849 log.cc:824] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:16.737885 7846 raft_consensus.cc:348] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.738345 7846 raft_consensus.cc:374] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:16.738454 7846 raft_consensus.cc:729] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8748a6950c4056bd5e326e4c250701, State: Initialized, Role: FOLLOWER
I20260927 15:07:16.739162 7846 consensus_queue.cc:260] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.739981 7849 tablet_bootstrap.cc:492] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: No bootstrap required, opened a new log
I20260927 15:07:16.740320 7812 heartbeater.cc:503] Master 127.0.97.62:34775 was elected leader, sending a full tablet report...
I20260927 15:07:16.740350 7849 ts_tablet_manager.cc:1397] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260927 15:07:16.743752 7846 ts_tablet_manager.cc:1428] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 15:07:16.748705 7845 raft_consensus.cc:348] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.749240 7845 raft_consensus.cc:374] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:16.749351 7845 raft_consensus.cc:729] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9c744a346d4c3fa599ef02298835ae, State: Initialized, Role: FOLLOWER
I20260927 15:07:16.750031 7845 consensus_queue.cc:260] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.751287 7670 heartbeater.cc:503] Master 127.0.97.62:34775 was elected leader, sending a full tablet report...
I20260927 15:07:16.752056 7845 ts_tablet_manager.cc:1428] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260927 15:07:16.754854 7849 raft_consensus.cc:348] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.755373 7849 raft_consensus.cc:374] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:16.755470 7849 raft_consensus.cc:729] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11403f429944d3d850732cf48ea6fff, State: Initialized, Role: FOLLOWER
I20260927 15:07:16.756146 7849 consensus_queue.cc:260] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.759042 7849 ts_tablet_manager.cc:1428] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Time spent starting tablet: real 0.019s user 0.010s sys 0.004s
W20260927 15:07:16.798635 7524 tablet.cc:2369] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:16.807525 7814 tablet.cc:2369] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:16.940431 7852 raft_consensus.cc:482] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:16.940810 7852 raft_consensus.cc:504] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.942612 7852 leader_election.cc:288] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795), d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133)
W20260927 15:07:16.948765 7674 tablet.cc:2369] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:16.950196 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.943187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39129 (local address 127.0.97.1:41133)
0927 15:07:16.943419 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.943428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.943929 (+ 501us) server_negotiation.cc:407] Connection header received
0927 15:07:16.944222 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.944227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.944288 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.944389 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:16.945010 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.945837 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.947914 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.948902 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.949103 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.949208 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.949809 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.949939 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:07:16.950369 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.942943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41133 (local address 127.0.97.2:39129)
0927 15:07:16.943772 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.943791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.944054 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.944416 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.944420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.944442 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:16.944829 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.944838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.945990 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.945998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.947713 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.947728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.949415 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.949421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.949626 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.949652 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.949788 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.949932 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":3,"thread_start_us":471,"threads_started":1}
I20260927 15:07:16.950546 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.942943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36665 (local address 127.0.97.3:43795)
0927 15:07:16.943196 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.943206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.944111 (+ 905us) server_negotiation.cc:407] Connection header received
0927 15:07:16.944323 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.944328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.944384 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.944483 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:16.945187 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.945920 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.948000 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.949551 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.949701 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.949815 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:16.950136 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 15:07:16.950271 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:07:16.951403 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.942982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43795 (local address 127.0.97.2:36665)
0927 15:07:16.943967 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:16.943988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:16.944179 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:16.944499 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:16.944503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:16.944591 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:16.944945 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.944954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.946058 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.946065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.947848 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:16.947860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.949813 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.949819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:16.949968 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.949992 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:16.950109 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:07:16.950258 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"spinlock_wait_cycles":14592,"thread_start_us":686,"threads_started":1}
I20260927 15:07:16.951553 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8346f23a6741487eac2c02051e79d2a7" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11403f429944d3d850732cf48ea6fff" is_pre_election: true
I20260927 15:07:16.952155 7476 raft_consensus.cc:2454] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 0.
I20260927 15:07:16.952639 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8346f23a6741487eac2c02051e79d2a7" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8748a6950c4056bd5e326e4c250701" is_pre_election: true
I20260927 15:07:16.953097 7557 leader_election.cc:302] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d11403f429944d3d850732cf48ea6fff, dc9c744a346d4c3fa599ef02298835ae; no voters:
I20260927 15:07:16.953255 7766 raft_consensus.cc:2454] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 0.
I20260927 15:07:16.953640 7852 raft_consensus.cc:2781] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:16.953797 7852 raft_consensus.cc:482] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:16.953900 7852 raft_consensus.cc:3037] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:16.956614 7852 raft_consensus.cc:504] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.957304 7852 leader_election.cc:288] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 election: Requested vote from peers 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795), d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133)
I20260927 15:07:16.957974 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8346f23a6741487eac2c02051e79d2a7" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8748a6950c4056bd5e326e4c250701"
I20260927 15:07:16.958029 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8346f23a6741487eac2c02051e79d2a7" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11403f429944d3d850732cf48ea6fff"
I20260927 15:07:16.958251 7476 raft_consensus.cc:3037] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:16.958244 7766 raft_consensus.cc:3037] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:16.960726 7766 raft_consensus.cc:2454] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 1.
I20260927 15:07:16.960724 7476 raft_consensus.cc:2454] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 1.
I20260927 15:07:16.961244 7558 leader_election.cc:302] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a8748a6950c4056bd5e326e4c250701, d11403f429944d3d850732cf48ea6fff, dc9c744a346d4c3fa599ef02298835ae; no voters:
I20260927 15:07:16.961660 7852 raft_consensus.cc:2781] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:16.962230 7852 raft_consensus.cc:686] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 1 LEADER]: Becoming Leader. State: Replica: dc9c744a346d4c3fa599ef02298835ae, State: Running, Role: LEADER
I20260927 15:07:16.962826 7852 consensus_queue.cc:237] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } }
I20260927 15:07:16.968750 6546 catalog_manager.cc:5495] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae reported cstate change: term changed from 0 to 1, leader changed from <none> to dc9c744a346d4c3fa599ef02298835ae (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dc9c744a346d4c3fa599ef02298835ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:17.000916 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:16.995252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.0.97.2:42201)
0927 15:07:16.995457 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:16.995495 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:16.995515 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:07:16.995617 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:16.995621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:16.995699 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:16.995837 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:16.996209 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.996987 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.998577 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:16.999004 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:16.999214 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:16.999747 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:17.000269 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:17.000386 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.000512 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.000601 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:07:17.009996 7476 raft_consensus.cc:1261] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Refusing update from remote peer dc9c744a346d4c3fa599ef02298835ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:17.009997 7766 raft_consensus.cc:1261] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Refusing update from remote peer dc9c744a346d4c3fa599ef02298835ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:17.011206 7856 consensus_queue.cc:1035] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 15:07:17.012060 7852 consensus_queue.cc:1035] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:17.025596 7860 mvcc.cc:205] Tried to move back new op lower bound from 7333976625181229056 to 7333976625003638784. Current Snapshot: MvccSnapshot[applied={T|T < 7333976625181229056}]
I20260927 15:07:17.044534 6546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:17.044881 6546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:17.054574 6546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54375309f80f4087988e32e540f70f37: Sending DeleteTablet for 3 replicas of tablet 8346f23a6741487eac2c02051e79d2a7
I20260927 15:07:17.055787 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 8346f23a6741487eac2c02051e79d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51570
I20260927 15:07:17.056077 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 8346f23a6741487eac2c02051e79d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35998
I20260927 15:07:17.059700 7871 tablet_replica.cc:330] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: stopping tablet replica
I20260927 15:07:17.060232 7871 raft_consensus.cc:2229] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:17.061290 7871 raft_consensus.cc:2258] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:17.062510 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 8346f23a6741487eac2c02051e79d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57488
I20260927 15:07:17.062167 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 8346f23a6741487eac2c02051e79d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35998
I20260927 15:07:17.065440 7872 tablet_replica.cc:330] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: stopping tablet replica
I20260927 15:07:17.066125 7872 raft_consensus.cc:2229] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:17.067108 7872 raft_consensus.cc:2258] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:17.067874 7871 ts_tablet_manager.cc:1905] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:17.068352 7870 tablet_replica.cc:330] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: stopping tablet replica
I20260927 15:07:17.068813 7870 raft_consensus.cc:2229] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:17.069360 7870 raft_consensus.cc:2258] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:17.070010 7872 ts_tablet_manager.cc:1905] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:17.072777 7870 ts_tablet_manager.cc:1905] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:17.075977 7871 ts_tablet_manager.cc:1918] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:17.076174 7871 log.cc:1196] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/8346f23a6741487eac2c02051e79d2a7
I20260927 15:07:17.076784 7871 ts_tablet_manager.cc:1939] T 8346f23a6741487eac2c02051e79d2a7 P d11403f429944d3d850732cf48ea6fff: Deleting consensus metadata
I20260927 15:07:17.078894 6530 catalog_manager.cc:4841] TS d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133): tablet 8346f23a6741487eac2c02051e79d2a7 (table test-table [id=29ec6cf6cc1b4458a67023799edf0ec8]) successfully deleted
I20260927 15:07:17.080397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 6514
W20260927 15:07:17.080400 6530 catalog_manager.cc:4808] TS d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133): delete failed for tablet 8346f23a6741487eac2c02051e79d2a7 because the tablet was not found. No further retry: Not found: Tablet not found: 8346f23a6741487eac2c02051e79d2a7
I20260927 15:07:17.080955 7870 ts_tablet_manager.cc:1918] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:17.081146 7870 log.cc:1196] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/8346f23a6741487eac2c02051e79d2a7
I20260927 15:07:17.081678 6576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:17.083701 7870 ts_tablet_manager.cc:1939] T 8346f23a6741487eac2c02051e79d2a7 P 2a8748a6950c4056bd5e326e4c250701: Deleting consensus metadata
I20260927 15:07:17.089236 7872 ts_tablet_manager.cc:1918] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:17.089497 7872 log.cc:1196] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/8346f23a6741487eac2c02051e79d2a7
I20260927 15:07:17.090271 7872 ts_tablet_manager.cc:1939] T 8346f23a6741487eac2c02051e79d2a7 P dc9c744a346d4c3fa599ef02298835ae: Deleting consensus metadata
I20260927 15:07:17.098821 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.064639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.97.61:34685)
0927 15:07:17.064841 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.064847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.070359 (+ 5512us) server_negotiation.cc:407] Connection header received
0927 15:07:17.070603 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.070609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.070702 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.070808 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.077978 (+ 7170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.079122 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.088137 (+ 9015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.088720 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.088894 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.092525 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.092532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.092535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.092575 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.095828 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.096608 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.096614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.096618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.096752 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.097334 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.097337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.097340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.097620 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.097886 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.098252 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.098480 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:07:17.112726 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.079676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.0.97.60:33959)
0927 15:07:17.079869 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.079875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.099025 (+ 19150us) server_negotiation.cc:407] Connection header received
0927 15:07:17.099343 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.099350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.099434 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.099504 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.101224 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.102121 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.103597 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.104008 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.104167 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.106772 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.106778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.106781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.106818 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.109802 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.110652 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.110662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.110665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.110794 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.111267 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.111273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.111277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.111568 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.111830 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.112259 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.112456 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:07:17.219826 6530 catalog_manager.cc:4841] TS dc9c744a346d4c3fa599ef02298835ae (127.0.97.2:42201): tablet 8346f23a6741487eac2c02051e79d2a7 (table test-table [id=29ec6cf6cc1b4458a67023799edf0ec8]) successfully deleted
I20260927 15:07:17.220177 6531 catalog_manager.cc:4841] TS 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795): tablet 8346f23a6741487eac2c02051e79d2a7 (table test-table [id=29ec6cf6cc1b4458a67023799edf0ec8]) successfully deleted
I20260927 15:07:17.220935 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 6514
I20260927 15:07:17.265928 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.249480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.0.97.60:33959)
0927 15:07:17.249699 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.249705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.249722 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:07:17.249799 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.249804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.249873 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.249969 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.252349 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.253277 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.254761 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.255482 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.255612 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.259155 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.259162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.259168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.259213 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.262802 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.263758 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.263764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.263768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.263913 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.264336 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.264341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.264345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.264716 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.264951 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.265362 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.265590 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:07:17.268160 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.247876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.97.61:34685)
0927 15:07:17.248067 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.248073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.248679 (+ 606us) server_negotiation.cc:407] Connection header received
0927 15:07:17.249031 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.249037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.249124 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.249201 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.254373 (+ 5172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.255331 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.257191 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.257854 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.258010 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.260928 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.260936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.260945 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.260980 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.264755 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.265607 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.265613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.265616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.265794 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.266273 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.266278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.266282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.266710 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.266921 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.267560 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.267788 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:07:18.038420 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.037603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:59769)
0927 15:07:18.038111 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.038254 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":276,"threads_started":1}
W20260927 15:07:18.038822 7812 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.042795 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.041996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:41677)
0927 15:07:18.042444 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.042557 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
W20260927 15:07:18.043399 7670 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.057297 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.056640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:36849)
0927 15:07:18.057033 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.057146 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":192,"threads_started":1}
W20260927 15:07:18.057740 7522 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.553782 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54375309f80f4087988e32e540f70f37)
I20260927 15:07:18.554131 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } }
I20260927 15:07:18.554687 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54375309f80f4087988e32e540f70f37)
I20260927 15:07:18.554939 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } }
I20260927 15:07:18.556059 7885 leader_election.cc:288] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775), 02ff713ce03f4875a4aaf3b237c52857 (127.0.97.61:34685)
I20260927 15:07:18.556488 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775), cfa1752c813343dbab208742746b3ca0 (127.0.97.60:33959)
I20260927 15:07:18.569877 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.556594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46738)
0927 15:07:18.569076 (+ 12482us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.569352 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12306,"thread_start_us":12136,"threads_started":1}
I20260927 15:07:18.572142 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.556696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46744)
0927 15:07:18.571466 (+ 14770us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.571697 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14630,"thread_start_us":11588,"threads_started":1}
W20260927 15:07:18.572077 7140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
W20260927 15:07:18.572494 7327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.576409 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.575823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46750)
0927 15:07:18.576050 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.576150 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260927 15:07:18.576678 7327 leader_election.cc:334] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.578325 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.576003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46758)
0927 15:07:18.577620 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.577797 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1481,"thread_start_us":1352,"threads_started":1}
W20260927 15:07:18.578603 7140 leader_election.cc:334] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.579888 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.556796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.97.61:34685)
0927 15:07:18.570706 (+ 13910us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.570712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.570731 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:07:18.570805 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.570811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.570901 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.572109 (+ 1208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:18.572579 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.574354 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.577555 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.578818 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.578971 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.579077 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.579396 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.579540 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13800,"thread_start_us":13608,"threads_started":1}
I20260927 15:07:18.579883 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.556292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34685 (local address 127.0.0.1:55566)
0927 15:07:18.567915 (+ 11623us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.567948 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:18.568146 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:18.571119 (+ 2973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:18.571127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:18.571175 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:18.571589 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.571611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.574654 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.574664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.577368 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.577385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.578989 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.578995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.579227 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.579252 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:07:18.579357 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:07:18.579527 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11411,"thread_start_us":14914,"threads_started":1}
I20260927 15:07:18.580773 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfa1752c813343dbab208742746b3ca0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02ff713ce03f4875a4aaf3b237c52857" is_pre_election: true
I20260927 15:07:18.581224 7163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfa1752c813343dbab208742746b3ca0 in term 1.
I20260927 15:07:18.581995 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.571681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33959 (local address 127.0.0.1:58604)
0927 15:07:18.571853 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.571870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:18.571980 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:18.575954 (+ 3974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:18.575961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:18.575991 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:18.576222 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.576233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.577864 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.577875 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.579770 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.579783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.581195 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.581203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.581401 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.581442 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:07:18.581568 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:07:18.581689 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:07:18.581960 7324 leader_election.cc:302] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ff713ce03f4875a4aaf3b237c52857, cfa1752c813343dbab208742746b3ca0; no voters: 54375309f80f4087988e32e540f70f37
I20260927 15:07:18.582477 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.569641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.0.97.60:33959)
0927 15:07:18.575092 (+ 5451us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.575104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.575129 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:07:18.575204 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.575211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.575323 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.575470 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:18.576438 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.577698 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.579945 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.580949 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.581116 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.581197 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.581649 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.581833 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5252,"thread_start_us":5071,"threads_started":1}
I20260927 15:07:18.582572 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:07:18.582705 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 54375309f80f4087988e32e540f70f37)
I20260927 15:07:18.582823 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:07:18.583160 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02ff713ce03f4875a4aaf3b237c52857" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cfa1752c813343dbab208742746b3ca0" is_pre_election: true
I20260927 15:07:18.585166 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } }
I20260927 15:07:18.585623 7350 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02ff713ce03f4875a4aaf3b237c52857 in current term 2: Already voted for candidate cfa1752c813343dbab208742746b3ca0 in this term.
I20260927 15:07:18.586009 7885 leader_election.cc:288] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [CANDIDATE]: Term 2 election: Requested vote from peers 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775), 02ff713ce03f4875a4aaf3b237c52857 (127.0.97.61:34685)
I20260927 15:07:18.586627 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.586140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46768)
0927 15:07:18.586320 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.586442 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:07:18.586679 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfa1752c813343dbab208742746b3ca0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02ff713ce03f4875a4aaf3b237c52857"
I20260927 15:07:18.586885 7140 leader_election.cc:302] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02ff713ce03f4875a4aaf3b237c52857; no voters: 54375309f80f4087988e32e540f70f37, cfa1752c813343dbab208742746b3ca0
I20260927 15:07:18.586905 7163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:07:18.587651 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.587308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46778)
0927 15:07:18.587462 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.587534 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:07:18.587872 7327 leader_election.cc:334] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
I20260927 15:07:18.589478 7163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfa1752c813343dbab208742746b3ca0 in term 2.
I20260927 15:07:18.589717 7886 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:07:18.590040 7324 leader_election.cc:302] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ff713ce03f4875a4aaf3b237c52857, cfa1752c813343dbab208742746b3ca0; no voters: 54375309f80f4087988e32e540f70f37
I20260927 15:07:18.590373 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:07:18.590863 7885 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [term 2 LEADER]: Becoming Leader. State: Replica: cfa1752c813343dbab208742746b3ca0, State: Running, Role: LEADER
I20260927 15:07:18.591334 7885 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } }
I20260927 15:07:18.593253 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa1752c813343dbab208742746b3ca0. Latest consensus state: current_term: 2 leader_uuid: "cfa1752c813343dbab208742746b3ca0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:18.593736 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:18.594491 7897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:18.598187 7897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:18.599061 7897 catalog_manager.cc:1259] Loaded cluster ID: 147140b2f02b47deaa8e84253542175e
I20260927 15:07:18.599238 7897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:18.600420 7897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:18.601241 7897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Loaded TSK: 0
I20260927 15:07:18.602123 7897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:18.633778 7340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:18.642580 7340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:18.647737 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.647299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46802)
0927 15:07:18.647506 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.647590 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:07:18.647706 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [term 2 FOLLOWER]: Refusing update from remote peer cfa1752c813343dbab208742746b3ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 15:07:18.648746 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 15:07:18.648988 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.648457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46818)
0927 15:07:18.648713 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.648812 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260927 15:07:18.649361 7327 consensus_peers.cc:487] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 -> Peer 54375309f80f4087988e32e540f70f37 (127.0.97.62:34775): Couldn't send request to peer 54375309f80f4087988e32e540f70f37. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:07:18.651800 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa1752c813343dbab208742746b3ca0. Latest consensus state: current_term: 2 leader_uuid: "cfa1752c813343dbab208742746b3ca0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:18.652099 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:18.653769 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "cfa1752c813343dbab208742746b3ca0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:18.654096 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:18.654736 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "cfa1752c813343dbab208742746b3ca0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "54375309f80f4087988e32e540f70f37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34775 } } peers { permanent_uuid: "02ff713ce03f4875a4aaf3b237c52857" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34685 } attrs { promote: false } } peers { permanent_uuid: "cfa1752c813343dbab208742746b3ca0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33959 } attrs { promote: false } } }
I20260927 15:07:18.654979 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ff713ce03f4875a4aaf3b237c52857 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:18.671324 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.663329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42201 (local address 127.0.0.1:57502)
0927 15:07:18.663495 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.663512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:18.663586 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:18.665165 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:18.665170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:18.665192 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:18.665385 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.665397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.666821 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.666826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.668726 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.668742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.670480 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.670486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.670674 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.670709 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:07:18.670816 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:07:18.670961 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 15:07:18.672176 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.663644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.97.2:42201)
0927 15:07:18.664834 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.664841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.664861 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:07:18.664933 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.664938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.665014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.665544 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:18.665749 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.666672 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.668940 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.670142 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.670287 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.670399 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.671625 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.671793 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":294,"threads_started":1}
I20260927 15:07:18.672549 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.663811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43795 (local address 127.0.0.1:51578)
0927 15:07:18.664449 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.664470 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:18.664580 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:18.665802 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:18.665810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:18.665836 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:18.666171 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.666184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.667753 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.667760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.669976 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.669988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.671557 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.671564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.671830 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.671869 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:07:18.671969 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:07:18.672110 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":27,"thread_start_us":319,"threads_started":1}
I20260927 15:07:18.673040 7604 tablet_service.cc:1467] Processing CreateTablet for tablet 96cd8941fd934e8bbb6a5b33383bf084 (DEFAULT_TABLE table=test-table [id=dd6de6ce0abf4875a66883fb692f8f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:18.673501 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cd8941fd934e8bbb6a5b33383bf084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:18.674441 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.664316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.0.97.3:43795)
0927 15:07:18.665378 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.665387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.665413 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:07:18.665503 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.665512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.665636 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.665777 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:18.666359 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.667613 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.670156 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.671310 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.671474 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.671603 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.672486 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.674181 (+ 1695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"spinlock_wait_cycles":9984,"thread_start_us":372,"threads_started":1}
I20260927 15:07:18.674919 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.662987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41133 (local address 127.0.0.1:36014)
0927 15:07:18.663183 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.663204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:18.663316 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:18.667828 (+ 4512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:18.667832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:18.667863 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:18.668293 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.668305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.669759 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.669765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.671858 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.671870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.674212 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.674218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.674389 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.674424 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:07:18.674530 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:07:18.674666 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":36}
I20260927 15:07:18.675205 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 96cd8941fd934e8bbb6a5b33383bf084 (DEFAULT_TABLE table=test-table [id=dd6de6ce0abf4875a66883fb692f8f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:18.675469 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.663156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.97.1:41133)
0927 15:07:18.667472 (+ 4316us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.667479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.667498 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:07:18.667566 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.667573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.667646 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.667807 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:18.668495 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.669607 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.672013 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.673819 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.673971 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.674063 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.674746 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.674893 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4179,"thread_start_us":4076,"threads_started":1}
I20260927 15:07:18.675664 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cd8941fd934e8bbb6a5b33383bf084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:18.676398 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 96cd8941fd934e8bbb6a5b33383bf084 (DEFAULT_TABLE table=test-table [id=dd6de6ce0abf4875a66883fb692f8f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:18.676821 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cd8941fd934e8bbb6a5b33383bf084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:18.679042 7908 tablet_bootstrap.cc:492] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Bootstrap starting.
I20260927 15:07:18.680671 7908 tablet_bootstrap.cc:654] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:18.682745 7908 tablet_bootstrap.cc:492] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: No bootstrap required, opened a new log
I20260927 15:07:18.682919 7908 ts_tablet_manager.cc:1397] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 15:07:18.683780 7908 raft_consensus.cc:348] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.684042 7908 raft_consensus.cc:374] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:18.684172 7908 raft_consensus.cc:729] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9c744a346d4c3fa599ef02298835ae, State: Initialized, Role: FOLLOWER
I20260927 15:07:18.684475 7908 consensus_queue.cc:260] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.690387 7910 tablet_bootstrap.cc:492] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Bootstrap starting.
I20260927 15:07:18.690775 7911 tablet_bootstrap.cc:492] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Bootstrap starting.
I20260927 15:07:18.692322 7908 ts_tablet_manager.cc:1428] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Time spent starting tablet: real 0.009s user 0.000s sys 0.009s
I20260927 15:07:18.694205 7910 tablet_bootstrap.cc:654] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:18.696380 7911 tablet_bootstrap.cc:654] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:18.698949 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.692147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:58719)
0927 15:07:18.698474 (+ 6327us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.698639 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6210,"thread_start_us":6104,"threads_started":1}
I20260927 15:07:18.700335 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.699893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:38325)
0927 15:07:18.700088 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.700189 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:07:18.701548 7669 heartbeater.cc:503] Master 127.0.97.60:33959 was elected leader, sending a full tablet report...
I20260927 15:07:18.701646 7911 tablet_bootstrap.cc:492] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: No bootstrap required, opened a new log
I20260927 15:07:18.701803 7911 ts_tablet_manager.cc:1397] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260927 15:07:18.702208 7910 tablet_bootstrap.cc:492] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: No bootstrap required, opened a new log
I20260927 15:07:18.702387 7910 ts_tablet_manager.cc:1397] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 15:07:18.702675 7911 raft_consensus.cc:348] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.702910 7911 raft_consensus.cc:374] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:18.703035 7911 raft_consensus.cc:729] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11403f429944d3d850732cf48ea6fff, State: Initialized, Role: FOLLOWER
I20260927 15:07:18.703310 7910 raft_consensus.cc:348] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.703337 7911 consensus_queue.cc:260] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.703585 7910 raft_consensus.cc:374] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:18.703691 7910 raft_consensus.cc:729] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8748a6950c4056bd5e326e4c250701, State: Initialized, Role: FOLLOWER
I20260927 15:07:18.703962 7910 consensus_queue.cc:260] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.704267 7911 ts_tablet_manager.cc:1428] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:18.705651 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.704556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:46077)
0927 15:07:18.705191 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.705348 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":334,"threads_started":1}
I20260927 15:07:18.706914 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.706465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:45473)
0927 15:07:18.706693 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.706778 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 15:07:18.708019 7521 heartbeater.cc:503] Master 127.0.97.60:33959 was elected leader, sending a full tablet report...
I20260927 15:07:18.710986 7910 ts_tablet_manager.cc:1428] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260927 15:07:18.712862 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.711788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:54631)
0927 15:07:18.712429 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.712559 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":324,"threads_started":1}
I20260927 15:07:18.713357 7811 heartbeater.cc:503] Master 127.0.97.60:33959 was elected leader, sending a full tablet report...
I20260927 15:07:18.714241 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.713865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:56073)
0927 15:07:18.714010 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.714100 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 15:07:18.733182 7912 raft_consensus.cc:482] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:18.733410 7912 raft_consensus.cc:504] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.734184 7912 leader_election.cc:288] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133), 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795)
I20260927 15:07:18.734747 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cd8941fd934e8bbb6a5b33383bf084" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11403f429944d3d850732cf48ea6fff" is_pre_election: true
I20260927 15:07:18.734804 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cd8941fd934e8bbb6a5b33383bf084" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8748a6950c4056bd5e326e4c250701" is_pre_election: true
I20260927 15:07:18.735045 7476 raft_consensus.cc:2454] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 0.
I20260927 15:07:18.735038 7766 raft_consensus.cc:2454] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 0.
I20260927 15:07:18.735577 7558 leader_election.cc:302] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a8748a6950c4056bd5e326e4c250701, d11403f429944d3d850732cf48ea6fff, dc9c744a346d4c3fa599ef02298835ae; no voters:
I20260927 15:07:18.735910 7912 raft_consensus.cc:2781] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:18.736045 7912 raft_consensus.cc:482] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:18.736177 7912 raft_consensus.cc:3037] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:18.738425 7912 raft_consensus.cc:504] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.739041 7912 leader_election.cc:288] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 election: Requested vote from peers d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133), 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795)
I20260927 15:07:18.739710 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cd8941fd934e8bbb6a5b33383bf084" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11403f429944d3d850732cf48ea6fff"
I20260927 15:07:18.739708 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cd8941fd934e8bbb6a5b33383bf084" candidate_uuid: "dc9c744a346d4c3fa599ef02298835ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8748a6950c4056bd5e326e4c250701"
I20260927 15:07:18.739943 7476 raft_consensus.cc:3037] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:18.739956 7766 raft_consensus.cc:3037] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:18.741992 7476 raft_consensus.cc:2454] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 1.
I20260927 15:07:18.741998 7766 raft_consensus.cc:2454] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9c744a346d4c3fa599ef02298835ae in term 1.
I20260927 15:07:18.742489 7557 leader_election.cc:302] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d11403f429944d3d850732cf48ea6fff, dc9c744a346d4c3fa599ef02298835ae; no voters:
I20260927 15:07:18.742856 7912 raft_consensus.cc:2781] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:18.743396 7912 raft_consensus.cc:686] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 1 LEADER]: Becoming Leader. State: Replica: dc9c744a346d4c3fa599ef02298835ae, State: Running, Role: LEADER
I20260927 15:07:18.743724 7912 consensus_queue.cc:237] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } }
I20260927 15:07:18.744571 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.744078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:50461)
0927 15:07:18.744241 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.744380 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:07:18.746831 7339 catalog_manager.cc:5495] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae reported cstate change: term changed from 0 to 1, leader changed from <none> to dc9c744a346d4c3fa599ef02298835ae (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dc9c744a346d4c3fa599ef02298835ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9c744a346d4c3fa599ef02298835ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:18.779946 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.775228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.97.2:42201)
0927 15:07:18.775488 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.775494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.775527 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:07:18.775616 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.775622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.775701 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.775802 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:18.776185 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.776862 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.778203 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.778635 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.778774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.779222 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:18.779341 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:18.779443 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.779588 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.779671 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:07:18.783241 7476 raft_consensus.cc:1261] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Refusing update from remote peer dc9c744a346d4c3fa599ef02298835ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:18.783447 7766 raft_consensus.cc:1261] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Refusing update from remote peer dc9c744a346d4c3fa599ef02298835ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:18.783959 7912 consensus_queue.cc:1035] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11403f429944d3d850732cf48ea6fff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:18.784204 7920 consensus_queue.cc:1035] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8748a6950c4056bd5e326e4c250701" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:18.787451 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.786393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:40269)
0927 15:07:18.786973 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.787212 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411}
I20260927 15:07:18.787832 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.787391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:36541)
0927 15:07:18.787550 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.787663 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:07:18.791452 7923 mvcc.cc:205] Tried to move back new op lower bound from 7333976632449622016 to 7333976632296636416. Current Snapshot: MvccSnapshot[applied={T|T < 7333976632449622016}]
I20260927 15:07:18.794126 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.793503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:36767)
0927 15:07:18.793880 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.793976 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267}
I20260927 15:07:18.794191 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.793503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.1:48423)
0927 15:07:18.793804 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.793948 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 15:07:18.795256 7925 mvcc.cc:205] Tried to move back new op lower bound from 7333976632449622016 to 7333976632296636416. Current Snapshot: MvccSnapshot[applied={T|T < 7333976632449622016}]
I20260927 15:07:18.798522 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.797835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.2:41911)
0927 15:07:18.798166 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.798314 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260927 15:07:18.802135 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.801644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.97.3:50995)
0927 15:07:18.801872 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.801963 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 15:07:18.802475 7339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:18.802845 7339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:18.810882 7339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfa1752c813343dbab208742746b3ca0: Sending DeleteTablet for 3 replicas of tablet 96cd8941fd934e8bbb6a5b33383bf084
I20260927 15:07:18.811936 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 96cd8941fd934e8bbb6a5b33383bf084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:18 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36014
I20260927 15:07:18.813230 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 96cd8941fd934e8bbb6a5b33383bf084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:18 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57502
I20260927 15:07:18.813424 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 96cd8941fd934e8bbb6a5b33383bf084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:18 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51578
I20260927 15:07:18.817121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 7377
I20260927 15:07:18.827316 7933 tablet_replica.cc:330] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: stopping tablet replica
I20260927 15:07:18.827625 7932 tablet_replica.cc:330] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: stopping tablet replica
I20260927 15:07:18.827762 7933 raft_consensus.cc:2229] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:18.827857 7932 raft_consensus.cc:2229] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:18.828136 7933 raft_consensus.cc:2258] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:18.828329 7932 raft_consensus.cc:2258] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:18.829674 7933 ts_tablet_manager.cc:1905] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:18.830035 7931 tablet_replica.cc:330] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: stopping tablet replica
I20260927 15:07:18.830259 7931 raft_consensus.cc:2229] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:18.830590 7931 raft_consensus.cc:2258] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:18.830773 7932 ts_tablet_manager.cc:1905] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:18.831964 7931 ts_tablet_manager.cc:1905] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:18.837900 7933 ts_tablet_manager.cc:1918] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:18.838078 7933 log.cc:1196] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/96cd8941fd934e8bbb6a5b33383bf084
I20260927 15:07:18.838601 7933 ts_tablet_manager.cc:1939] T 96cd8941fd934e8bbb6a5b33383bf084 P 2a8748a6950c4056bd5e326e4c250701: Deleting consensus metadata
I20260927 15:07:18.839213 7931 ts_tablet_manager.cc:1918] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:18.839366 7931 log.cc:1196] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/96cd8941fd934e8bbb6a5b33383bf084
I20260927 15:07:18.839865 7931 ts_tablet_manager.cc:1939] T 96cd8941fd934e8bbb6a5b33383bf084 P d11403f429944d3d850732cf48ea6fff: Deleting consensus metadata
I20260927 15:07:18.839978 7325 catalog_manager.cc:4841] TS 2a8748a6950c4056bd5e326e4c250701 (127.0.97.3:43795): tablet 96cd8941fd934e8bbb6a5b33383bf084 (table test-table [id=dd6de6ce0abf4875a66883fb692f8f85]) successfully deleted
I20260927 15:07:18.841014 7324 catalog_manager.cc:4841] TS d11403f429944d3d850732cf48ea6fff (127.0.97.1:41133): tablet 96cd8941fd934e8bbb6a5b33383bf084 (table test-table [id=dd6de6ce0abf4875a66883fb692f8f85]) successfully deleted
I20260927 15:07:18.841213 7932 ts_tablet_manager.cc:1918] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:18.841380 7932 log.cc:1196] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/96cd8941fd934e8bbb6a5b33383bf084
I20260927 15:07:18.841845 7932 ts_tablet_manager.cc:1939] T 96cd8941fd934e8bbb6a5b33383bf084 P dc9c744a346d4c3fa599ef02298835ae: Deleting consensus metadata
I20260927 15:07:18.842921 7324 catalog_manager.cc:4841] TS dc9c744a346d4c3fa599ef02298835ae (127.0.97.2:42201): tablet 96cd8941fd934e8bbb6a5b33383bf084 (table test-table [id=dd6de6ce0abf4875a66883fb692f8f85]) successfully deleted
I20260927 15:07:18.847772 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.818366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.0.97.1:41133)
0927 15:07:18.818662 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.818672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.829265 (+ 10593us) server_negotiation.cc:407] Connection header received
0927 15:07:18.829676 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.829683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.829800 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.829960 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:18.831943 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.833207 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.835186 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.835997 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.836195 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.840463 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:18.840484 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:18.840505 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:18.840562 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:18.844529 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.845444 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.845454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.845457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.845609 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.846066 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.846071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.846075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.846529 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:18.846856 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.847216 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.847472 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:07:18.848531 7516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:18.942641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 7377
I20260927 15:07:18.961853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 7533
I20260927 15:07:18.976881 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.962786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.0.97.2:42201)
0927 15:07:18.962990 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.962999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.963279 (+ 280us) server_negotiation.cc:407] Connection header received
0927 15:07:18.963510 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.963516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.963601 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.963713 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:18.965054 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.965778 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.967133 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.967572 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.967721 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.970426 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:18.970441 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:18.970458 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:18.970499 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:18.973554 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.974353 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.974370 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.974374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.974515 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.974955 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.974960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.974965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.975631 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:18.975961 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.976286 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.976523 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:07:18.977568 7664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:19.025411 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.024942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46824)
0927 15:07:19.025149 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:19.025257 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:07:19.026499 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.026157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34775 (local address 127.0.0.1:46836)
0927 15:07:19.026310 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:19.026388 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:07:19.099872 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 7533
I20260927 15:07:19.122920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 7682
I20260927 15:07:19.139642 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.123790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.0.97.3:43795)
0927 15:07:19.124002 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:19.124006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:19.124025 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:07:19.124200 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:19.124205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:19.124302 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:19.124397 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:19.126174 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.126906 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.128934 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.129412 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.129525 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:19.132846 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:19.132861 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:19.132878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:19.132935 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:19.136290 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.137416 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.137424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.137427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.137582 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.138069 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.138073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.138075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.138477 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:19.138732 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:19.139050 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 15:07:19.139346 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:07:19.140316 7806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:19.236253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 7682
I20260927 15:07:19.255548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 7009
I20260927 15:07:19.256783 7183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:19.376307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 7009
I20260927 15:07:19.397473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 7109
I20260927 15:07:19.398696 7370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:19.521565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 7109
2026-09-27T15:07:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9914 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:19 dist-test-slave-g0w1 krb5kdc[7942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:19 dist-test-slave-g0w1 krb5kdc[7942](info): set up 2 sockets
Sep 27 15:07:19 dist-test-slave-g0w1 krb5kdc[7942](info): commencing operation
krb5kdc: starting...
W20260927 15:07:20.640105 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:20 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 15:07:20.875959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43843
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:21.214111 7955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:21.214426 7955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:21.214601 7955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:21.222185 7955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:21.222306 7955 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:07:21.222384 7955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:21.222484 7955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:21.222568 7955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:21.222677 7955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:21.233031 7955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43843
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:21.234426 7955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:21.235853 7955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:21.248109 7960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:21.248040 7961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:21.248224 7963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:21.249189 7955 server_base.cc:1034] running on GCE node
I20260927 15:07:21.249962 7955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:21.251426 7955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:21.252651 7955 hybrid_clock.cc:648] HybridClock initialized: now 1790521641252570 us; error 53 us; skew 500 ppm
I20260927 15:07:21.253003 7955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:21 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:21.256309 7955 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 15:07:21.257928 7955 webserver.cc:460] Webserver started at http://127.0.97.62:42605/ using document root <none> and password file <none>
I20260927 15:07:21.258648 7955 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:21.258792 7955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:21.259174 7955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:21.262181 7955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "4babd43ee713449f8d4e774deeb47fad"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-g0w1"
server_key: "6d45afcd7b4c5e787741320413ea8b8c"
server_key_iv: "df5dea2bc01096308702ad88157300fd"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.263016 7955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "4babd43ee713449f8d4e774deeb47fad"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-g0w1"
server_key: "6d45afcd7b4c5e787741320413ea8b8c"
server_key_iv: "df5dea2bc01096308702ad88157300fd"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.268584 7955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 15:07:21.272331 7970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:21.273859 7955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:07:21.274044 7955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "4babd43ee713449f8d4e774deeb47fad"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-g0w1"
server_key: "6d45afcd7b4c5e787741320413ea8b8c"
server_key_iv: "df5dea2bc01096308702ad88157300fd"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.274286 7955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:21.300005 7955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:21.304626 7955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:21.304942 7955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:21.327340 7955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43843
I20260927 15:07:21.327368 8031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43843 every 8 connection(s)
I20260927 15:07:21.328775 7955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:21.332836 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:21.339308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 7955
I20260927 15:07:21.339563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:21.340389 388 external_mini_cluster.cc:1428] Setting key 476f85e7516674525d6b182e39c0a1a6
I20260927 15:07:21.344887 8032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Bootstrap starting.
I20260927 15:07:21.348953 8032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:21.350246 8032 log.cc:824] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:21.353856 8032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: No bootstrap required, opened a new log
Sep 27 15:07:21 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:21.362238 8032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4babd43ee713449f8d4e774deeb47fad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43843 } }
I20260927 15:07:21.362695 8032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:21.362833 8032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4babd43ee713449f8d4e774deeb47fad, State: Initialized, Role: FOLLOWER
I20260927 15:07:21.363656 8032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4babd43ee713449f8d4e774deeb47fad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43843 } }
I20260927 15:07:21.364099 8032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:21.364280 8032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:21.364470 8032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:21.365325 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.342731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.0.97.62:43843)
0927 15:07:21.343613 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:21.343626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:21.343670 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:07:21.345447 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:21.345472 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:21.345888 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:21.346245 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:21.348236 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.349719 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.351629 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.352372 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.352536 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:21.357376 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:21.357384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:21.357405 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:21.357451 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:21.360793 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.361556 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.361563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.361570 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.361751 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.362149 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.362158 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.362160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.362823 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:21.363219 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:21.363614 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 15:07:21.364055 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":435,"threads_started":1}
I20260927 15:07:21.367169 8032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4babd43ee713449f8d4e774deeb47fad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43843 } }
I20260927 15:07:21.367851 8032 leader_election.cc:302] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4babd43ee713449f8d4e774deeb47fad; no voters:
I20260927 15:07:21.368619 8032 leader_election.cc:288] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:21.368760 8037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:21.369580 8037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [term 1 LEADER]: Becoming Leader. State: Replica: 4babd43ee713449f8d4e774deeb47fad, State: Running, Role: LEADER
I20260927 15:07:21.370196 8037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4babd43ee713449f8d4e774deeb47fad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43843 } }
I20260927 15:07:21.370944 8032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:21.373284 8039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4babd43ee713449f8d4e774deeb47fad. Latest consensus state: current_term: 1 leader_uuid: "4babd43ee713449f8d4e774deeb47fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4babd43ee713449f8d4e774deeb47fad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43843 } } }
I20260927 15:07:21.373768 8039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:21.374871 8038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4babd43ee713449f8d4e774deeb47fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4babd43ee713449f8d4e774deeb47fad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43843 } } }
I20260927 15:07:21.375303 8038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:21.381692 8042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:21.386128 8042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:21.394936 8042 catalog_manager.cc:1347] Generated new cluster ID: b8b103f0b6694824976ed06d9f3e7fa0
I20260927 15:07:21.395113 8042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:21.409691 8042 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:21.411064 8042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:21.417045 8042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Generated new TSK 0
I20260927 15:07:21.417964 8042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 15:07:21.483548 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43843
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:21.835932 8058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:21.836246 8058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:21.836423 8058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:21.844017 8058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:21.844142 8058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:21.844304 8058 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:21.844439 8058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:21.854504 8058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:21.855882 8058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:21.857286 8058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:21.871770 8064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:21.871896 8066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:21.872421 8063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:21.878553 8058 server_base.cc:1034] running on GCE node
I20260927 15:07:21.879405 8058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:21.880445 8058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:21.881805 8058 hybrid_clock.cc:648] HybridClock initialized: now 1790521641881740 us; error 59 us; skew 500 ppm
I20260927 15:07:21.882174 8058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:21 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:21.885774 8058 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 15:07:21.887674 8058 webserver.cc:460] Webserver started at http://127.0.97.1:33085/ using document root <none> and password file <none>
I20260927 15:07:21.888523 8058 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:21.888669 8058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:21.889039 8058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:21.892287 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "1913adf0f67c403e9667952d11624878"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-g0w1"
server_key: "eb4a2e8b0c99f183f9b4cc6aae24aadd"
server_key_iv: "1b2da484ae73659bf8320333ef5482d5"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.893152 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1913adf0f67c403e9667952d11624878"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-g0w1"
server_key: "eb4a2e8b0c99f183f9b4cc6aae24aadd"
server_key_iv: "1b2da484ae73659bf8320333ef5482d5"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.898644 8058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:07:21.902686 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:21.904363 8058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:21.904573 8058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "1913adf0f67c403e9667952d11624878"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-g0w1"
server_key: "eb4a2e8b0c99f183f9b4cc6aae24aadd"
server_key_iv: "1b2da484ae73659bf8320333ef5482d5"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.904832 8058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:21.920778 8058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:21.925390 8058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:21.925697 8058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:21.928670 8058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:21.928793 8058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:21.928953 8058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:21.929059 8058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:21 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:21.957176 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.930156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.97.1:58043)
0927 15:07:21.931132 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:21.931196 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:21.933218 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:21.934232 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:21.934245 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:21.935242 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:21.936258 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:21.936284 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.938162 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.938173 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:21.940340 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:21.940356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.941550 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.941562 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:21.942077 (+ 515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:21.943061 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:21.943134 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:21.947388 (+ 4254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:21.951575 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:21.951585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:21.951613 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:21.952299 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:21.952864 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:21.952874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:21.952879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:21.953387 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:21.954179 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:21.954190 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:21.954697 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 15:07:21.955150 (+ 453us) client_negotiation.cc:240] Negotiation successful
0927 15:07:21.955892 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":376,"threads_started":1}
I20260927 15:07:21.958454 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.929362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58043 (local address 127.0.97.62:43843)
0927 15:07:21.930056 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:21.930063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:21.931342 (+ 1279us) server_negotiation.cc:407] Connection header received
0927 15:07:21.933717 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:21.933723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:21.933821 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:21.933958 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:21.936568 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.937926 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.940543 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.941188 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.941388 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:21.947713 (+ 6325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:21.947719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:21.947722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:21.947784 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:21.951384 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.952494 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.952501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.952504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.952703 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.953564 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.953571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.953576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.953980 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:21.954159 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:21.957936 (+ 3777us) server_negotiation.cc:299] Negotiation successful
0927 15:07:21.958191 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":452,"threads_started":1}
I20260927 15:07:21.992872 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.974140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43665 (local address 127.0.97.62:43843)
0927 15:07:21.974374 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:21.974379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:21.975414 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 15:07:21.975745 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:21.975754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:21.975871 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:21.976013 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:21.979024 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.980385 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.981847 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.982813 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.982995 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:21.985020 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:21.985029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:21.985037 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:21.985082 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:21.989164 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.990181 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.990189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.990195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.990367 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.990906 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.990911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.990914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.991341 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:21.991523 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:21.992257 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 15:07:21.992498 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 15:07:21.993037 8058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46585
I20260927 15:07:21.993078 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.974223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.97.1:43665)
0927 15:07:21.975221 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:21.975244 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:21.975543 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:21.976471 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:21.976476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:21.978421 (+ 1945us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:21.978836 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:21.978852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.980578 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.980585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:21.981675 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:21.981688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.983038 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.983043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:21.983287 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:21.984014 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:21.984032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:21.984782 (+ 750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:21.989393 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:21.989405 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:21.989412 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:21.990037 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:21.990557 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:21.990565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:21.990570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:21.990776 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:21.991519 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:21.991526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:21.991781 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 15:07:21.991980 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 15:07:21.992489 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":498,"threads_started":1}
I20260927 15:07:21.993163 8193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46585 every 8 connection(s)
I20260927 15:07:21.994632 8058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:21.997080 8194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43843
I20260927 15:07:21.997344 8194 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:21.998112 8194 heartbeater.cc:511] Master 127.0.97.62:43843 requested a full tablet report, sending...
I20260927 15:07:21.998575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 8058
I20260927 15:07:21.998793 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:21.999677 388 external_mini_cluster.cc:1428] Setting key c16004a126b3dba9d39ee640840e80f7
I20260927 15:07:22.000702 7987 ts_manager.cc:194] Registered new tserver with Master: 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585)
I20260927 15:07:22.002504 7987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58043
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 15:07:22.051821 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43843
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:22.386734 8198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:22.387036 8198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:22.387274 8198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:22.394884 8198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:22.395009 8198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:22.395181 8198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:22.395349 8198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:22.405515 8198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:22.406862 8198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:22.408295 8198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:22.422062 8204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:22.421954 8203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:22.422549 8206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:22.423202 8198 server_base.cc:1034] running on GCE node
I20260927 15:07:22.423904 8198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:22.424975 8198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:22.426257 8198 hybrid_clock.cc:648] HybridClock initialized: now 1790521642426181 us; error 57 us; skew 500 ppm
I20260927 15:07:22.426587 8198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:22 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:22.430167 8198 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 15:07:22.431910 8198 webserver.cc:460] Webserver started at http://127.0.97.2:43535/ using document root <none> and password file <none>
I20260927 15:07:22.432668 8198 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:22.432812 8198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:22.433149 8198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:22.436225 8198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "6336946210c5435588a37d69d3160037"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-g0w1"
server_key: "2cc1cd9b64ab0c7cbf8576fd3fdd7d69"
server_key_iv: "5e4a11aebe0434b6973f2851f0af2db0"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.437031 8198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6336946210c5435588a37d69d3160037"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-g0w1"
server_key: "2cc1cd9b64ab0c7cbf8576fd3fdd7d69"
server_key_iv: "5e4a11aebe0434b6973f2851f0af2db0"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.442370 8198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:07:22.446120 8213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.447556 8198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:22.447885 8198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "6336946210c5435588a37d69d3160037"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-g0w1"
server_key: "2cc1cd9b64ab0c7cbf8576fd3fdd7d69"
server_key_iv: "5e4a11aebe0434b6973f2851f0af2db0"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.448159 8198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:22.478053 8198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:22.482970 8198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:22.483297 8198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:22.486217 8198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:22.486334 8198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.486495 8198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:22.486582 8198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:22 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:22.514647 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.486897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57721 (local address 127.0.97.62:43843)
0927 15:07:22.487125 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.487131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.488867 (+ 1736us) server_negotiation.cc:407] Connection header received
0927 15:07:22.492204 (+ 3337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.492213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.492326 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.492461 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.495527 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.496568 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.499051 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.499751 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.499914 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.505699 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.505709 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.505716 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.505768 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.509549 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.511036 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.511043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.511046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.511257 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.512218 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.512224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.512228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.512611 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.512792 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.514074 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.514358 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:07:22.516881 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.487673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.97.2:57721)
0927 15:07:22.488666 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.488724 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.491711 (+ 2987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.492810 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.492824 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.493905 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.495178 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.495214 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.496834 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.496841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.498820 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.498841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.500057 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.500073 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.500492 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.501398 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.501442 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.505380 (+ 3938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.509794 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.509808 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.509841 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.510782 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.511421 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.511429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.511439 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.511921 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.512808 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.512823 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.513509 (+ 686us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.513940 (+ 431us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.514727 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":420,"threads_started":1}
I20260927 15:07:22.539177 8198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46245
I20260927 15:07:22.539285 8332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46245 every 8 connection(s)
I20260927 15:07:22.539479 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.523321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54095 (local address 127.0.97.62:43843)
0927 15:07:22.523495 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.523500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.524204 (+ 704us) server_negotiation.cc:407] Connection header received
0927 15:07:22.524437 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.524446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.524545 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.524649 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.525666 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526437 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.529262 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.529667 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.529776 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.532655 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.532664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.532671 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.532720 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.536428 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.537374 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.537382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.537390 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.537509 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.537907 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.537916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.537918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.538247 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.538402 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.538832 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.539057 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:07:22.539762 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.523394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.97.2:54095)
0927 15:07:22.524077 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.524101 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.524282 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.524786 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.524790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.525222 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.525508 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.525518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526620 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.526627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.529086 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.529108 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.529882 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.529890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.530105 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.530894 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.530923 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.532400 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.536598 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.536604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.536608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.537232 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.537634 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.537639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.537643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.537783 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.538370 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.538375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.538566 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.538731 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.539124 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":379,"threads_started":1}
I20260927 15:07:22.540911 8198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:22.543373 8333 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43843
I20260927 15:07:22.543633 8333 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:22.544327 8333 heartbeater.cc:511] Master 127.0.97.62:43843 requested a full tablet report, sending...
I20260927 15:07:22.545167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 8198
I20260927 15:07:22.545379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:22.545740 7987 ts_manager.cc:194] Registered new tserver with Master: 6336946210c5435588a37d69d3160037 (127.0.97.2:46245)
I20260927 15:07:22.545878 388 external_mini_cluster.cc:1428] Setting key 06ebe7b14e81265695af5cd715f75743
I20260927 15:07:22.546639 7987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57721
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 15:07:22.595252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43843
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:22.945400 8337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:22.945714 8337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:22.945896 8337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:22.953507 8337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:22.953644 8337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:22.953728 8337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:22.953809 8337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:22.964061 8337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:22.965387 8337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:22.966856 8337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:22.982321 8342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:22.982334 8343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:22.983135 8345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:22.985044 8337 server_base.cc:1034] running on GCE node
I20260927 15:07:22.985808 8337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:22.986948 8337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:22.988236 8337 hybrid_clock.cc:648] HybridClock initialized: now 1790521642988120 us; error 50 us; skew 500 ppm
I20260927 15:07:22.988531 8337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:22 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:22.992067 8337 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 15:07:22.994048 8337 webserver.cc:460] Webserver started at http://127.0.97.3:35381/ using document root <none> and password file <none>
I20260927 15:07:22.995028 8337 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:22.995205 8337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:22.995582 8337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:22.999686 8337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb6b150470e746e6801fb8cdddfdf310"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-g0w1"
server_key: "e75f937d2efe3d2606e7df165407aea3"
server_key_iv: "23d48501e448f2fd8b01ae43082fa3d7"
server_key_version: "encryptionkey@0"
I20260927 15:07:23.000516 8337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb6b150470e746e6801fb8cdddfdf310"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-g0w1"
server_key: "e75f937d2efe3d2606e7df165407aea3"
server_key_iv: "23d48501e448f2fd8b01ae43082fa3d7"
server_key_version: "encryptionkey@0"
I20260927 15:07:23.006111 8337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:07:23.006280 8194 heartbeater.cc:503] Master 127.0.97.62:43843 was elected leader, sending a full tablet report...
I20260927 15:07:23.010808 8352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:23.012631 8337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:23.012846 8337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "cb6b150470e746e6801fb8cdddfdf310"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-g0w1"
server_key: "e75f937d2efe3d2606e7df165407aea3"
server_key_iv: "23d48501e448f2fd8b01ae43082fa3d7"
server_key_version: "encryptionkey@0"
I20260927 15:07:23.013104 8337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:23.042248 8337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:23.047407 8337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:23.047716 8337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:23.050551 8337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:23.050712 8337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:23.050876 8337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:23.050954 8337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:23 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:23.076650 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.051497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55941 (local address 127.0.97.62:43843)
0927 15:07:23.051919 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.051928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.053810 (+ 1882us) server_negotiation.cc:407] Connection header received
0927 15:07:23.056512 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.056518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.056619 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.056763 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.059432 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.060517 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.063185 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.063629 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.063794 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.069056 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:23.069061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:23.069065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:23.069100 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:23.072394 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.073542 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.073548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.073552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.073708 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.074266 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.074273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.074277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.074782 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:23.074973 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.076014 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.076255 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":1476,"threads_started":1}
I20260927 15:07:23.077872 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.052630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.97.3:55941)
0927 15:07:23.053586 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.053650 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.056088 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.057076 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.057092 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.058103 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:23.059216 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.059242 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.060700 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.060708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.062951 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.062967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.063903 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.063912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.064347 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.065291 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:23.065364 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:23.068731 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:23.072633 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:23.072645 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:23.072694 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:23.073386 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:23.073835 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:23.073840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:23.073844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:23.074069 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:23.074956 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:23.074967 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:23.075485 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.075901 (+ 416us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.076505 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":427,"threads_started":1}
I20260927 15:07:23.103453 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.085357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37627 (local address 127.0.97.62:43843)
0927 15:07:23.085601 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.085608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.086526 (+ 918us) server_negotiation.cc:407] Connection header received
0927 15:07:23.086803 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.086829 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.086932 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.087051 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.088848 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.089595 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.092379 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.093060 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.093229 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.096124 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:23.096129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:23.096132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:23.096194 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:23.099901 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.101023 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.101027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.101029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.101199 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.101744 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.101750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.101755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.102110 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:23.102308 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.102910 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.103202 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:07:23.104009 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.085421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.97.3:37627)
0927 15:07:23.086346 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.086376 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.086614 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.087135 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.087142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.088007 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:23.088647 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.088668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.089773 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.089783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.092113 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.092134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.093531 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.093550 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.093795 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.094664 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:23.094695 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:23.095839 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:23.100164 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:23.100173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:23.100180 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:23.100841 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:23.101351 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:23.101358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:23.101363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:23.101578 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:23.102264 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:23.102270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:23.102577 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.102769 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.103285 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":536,"threads_started":1}
I20260927 15:07:23.109587 8337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37971
I20260927 15:07:23.109751 8472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37971 every 8 connection(s)
I20260927 15:07:23.110862 8337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:23.113222 8473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43843
I20260927 15:07:23.113433 8473 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:23.114086 8473 heartbeater.cc:511] Master 127.0.97.62:43843 requested a full tablet report, sending...
I20260927 15:07:23.115751 7986 ts_manager.cc:194] Registered new tserver with Master: cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:23.116658 7986 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55941
I20260927 15:07:23.119407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 8337
I20260927 15:07:23.119630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:23.120042 388 external_mini_cluster.cc:1428] Setting key cd75b95704d4170c2ccdf53c7e2d8489
I20260927 15:07:23.123690 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:07:23 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:07:23 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:23 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:23.178314 8361 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:23.166133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.0.97.62:43843)
0927 15:07:23.166395 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.166402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.166722 (+ 320us) server_negotiation.cc:407] Connection header received
0927 15:07:23.166920 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.166925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.166994 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.167224 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.168683 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.169454 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.170826 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.171484 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.171617 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.177964 (+ 6347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46344: BlockingRecv error: recv got EOF from 127.0.0.1:46344 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 15:07:23 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:23.196520 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.181812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.97.62:43843)
0927 15:07:23.181989 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.181994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.182447 (+ 453us) server_negotiation.cc:407] Connection header received
0927 15:07:23.182593 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.182597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.182660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.182771 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.184050 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.184781 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.186150 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.186567 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.186681 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.190831 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:23.190836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:23.190840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:23.190881 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:23.193631 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.194418 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.194424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.194427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.194565 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.194980 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.194986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.194989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.195354 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:23.195621 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.195923 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.196159 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:07:23.201017 7986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:23.208443 7986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:23.252140 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.222455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.0.97.2:46245)
0927 15:07:23.237992 (+ 15537us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.238008 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.240385 (+ 2377us) server_negotiation.cc:407] Connection header received
0927 15:07:23.240501 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.240513 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.240833 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.241044 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:23.242190 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.244273 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.249132 (+ 4859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.250740 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.250870 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.251142 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.251669 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.251844 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":15569,"threads_started":1}
I20260927 15:07:23.252239 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.222477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46245 (local address 127.0.0.1:49096)
0927 15:07:23.239264 (+ 16787us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.239336 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.239619 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.241314 (+ 1695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.241322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.241353 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:23.241927 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.241946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.245436 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.245444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.248829 (+ 3385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.248858 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.251048 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.251057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.251365 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.251442 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.251602 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.251819 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16376,"thread_start_us":26761,"threads_started":1}
I20260927 15:07:23.256757 8263 tablet_service.cc:1467] Processing CreateTablet for tablet d85a01406e2946a790a1aceaa3c9c18e (DEFAULT_TABLE table=test-table [id=ffca7b2823204ee29de72eaa7d2a77ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:23.258275 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85a01406e2946a790a1aceaa3c9c18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.258553 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.222851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.0.97.3:37971)
0927 15:07:23.249992 (+ 27141us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.250008 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.250035 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:23.250109 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.250114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.250308 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.250540 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:23.251378 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.253251 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.255737 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.257279 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.257418 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.257689 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.258027 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.258197 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26949,"thread_start_us":26755,"threads_started":1}
I20260927 15:07:23.258637 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.224901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46585 (local address 127.0.0.1:39916)
0927 15:07:23.249517 (+ 24616us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.249542 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.249814 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.250789 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.250801 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.250835 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:23.251516 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.251531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.253482 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.253485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.255782 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.255798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.257824 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.257829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.257990 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.258023 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.258098 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.258243 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24471,"mutex_wait_us":4,"thread_start_us":13790,"threads_started":1}
I20260927 15:07:23.259162 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.222524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37971 (local address 127.0.0.1:58452)
0927 15:07:23.245081 (+ 22557us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.245110 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.245267 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.250522 (+ 5255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.250528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.250556 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:23.251139 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.251150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.253412 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.253418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.255548 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.255563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.257592 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.257599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.257794 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.257839 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.257934 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.258083 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22363,"mutex_wait_us":30,"thread_start_us":22111,"threads_started":1}
I20260927 15:07:23.259343 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.225553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.97.1:46585)
0927 15:07:23.240307 (+ 14754us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.240316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.249676 (+ 9360us) server_negotiation.cc:407] Connection header received
0927 15:07:23.250007 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.250017 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.250331 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.250557 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:23.251791 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.253298 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.255965 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.257392 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.257503 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.257674 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.258870 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.259003 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14586,"thread_start_us":14428,"threads_started":1}
I20260927 15:07:23.262771 8403 tablet_service.cc:1467] Processing CreateTablet for tablet d85a01406e2946a790a1aceaa3c9c18e (DEFAULT_TABLE table=test-table [id=ffca7b2823204ee29de72eaa7d2a77ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:23.263027 8124 tablet_service.cc:1467] Processing CreateTablet for tablet d85a01406e2946a790a1aceaa3c9c18e (DEFAULT_TABLE table=test-table [id=ffca7b2823204ee29de72eaa7d2a77ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:23.264358 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85a01406e2946a790a1aceaa3c9c18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.264408 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85a01406e2946a790a1aceaa3c9c18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.276263 8498 tablet_bootstrap.cc:492] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Bootstrap starting.
I20260927 15:07:23.280323 8498 tablet_bootstrap.cc:654] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.281015 8497 tablet_bootstrap.cc:492] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Bootstrap starting.
I20260927 15:07:23.281759 8499 tablet_bootstrap.cc:492] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Bootstrap starting.
I20260927 15:07:23.281752 8498 log.cc:824] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:23.285353 8497 tablet_bootstrap.cc:654] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.285475 8499 tablet_bootstrap.cc:654] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.286268 8498 tablet_bootstrap.cc:492] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: No bootstrap required, opened a new log
I20260927 15:07:23.286540 8497 log.cc:824] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:23.286582 8498 ts_tablet_manager.cc:1397] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 15:07:23.286722 8499 log.cc:824] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:23.290149 8497 tablet_bootstrap.cc:492] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: No bootstrap required, opened a new log
I20260927 15:07:23.290469 8497 ts_tablet_manager.cc:1397] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260927 15:07:23.294456 8499 tablet_bootstrap.cc:492] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: No bootstrap required, opened a new log
I20260927 15:07:23.294801 8499 ts_tablet_manager.cc:1397] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260927 15:07:23.298116 8497 raft_consensus.cc:348] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.298557 8497 raft_consensus.cc:374] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.298663 8497 raft_consensus.cc:729] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6336946210c5435588a37d69d3160037, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.298730 8498 raft_consensus.cc:348] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.299238 8498 raft_consensus.cc:374] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.299342 8498 raft_consensus.cc:729] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6b150470e746e6801fb8cdddfdf310, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.299965 8498 consensus_queue.cc:260] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.299481 8497 consensus_queue.cc:260] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.301504 8333 heartbeater.cc:503] Master 127.0.97.62:43843 was elected leader, sending a full tablet report...
I20260927 15:07:23.302083 8497 ts_tablet_manager.cc:1428] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260927 15:07:23.307214 8499 raft_consensus.cc:348] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.307890 8499 raft_consensus.cc:374] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.308022 8499 raft_consensus.cc:729] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1913adf0f67c403e9667952d11624878, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.308815 8473 heartbeater.cc:503] Master 127.0.97.62:43843 was elected leader, sending a full tablet report...
I20260927 15:07:23.308858 8499 consensus_queue.cc:260] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.310105 8498 ts_tablet_manager.cc:1428] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Time spent starting tablet: real 0.023s user 0.010s sys 0.012s
I20260927 15:07:23.311058 8499 ts_tablet_manager.cc:1428] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260927 15:07:23.362623 8474 tablet.cc:2369] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:23.427232 8504 raft_consensus.cc:482] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:23.427598 8504 raft_consensus.cc:504] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.429322 8504 leader_election.cc:288] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971), 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585)
I20260927 15:07:23.436854 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.429852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58531 (local address 127.0.97.3:37971)
0927 15:07:23.430076 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.430085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.430887 (+ 802us) server_negotiation.cc:407] Connection header received
0927 15:07:23.431109 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.431118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.431207 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.431280 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:23.431968 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.432708 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.434743 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.435685 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.435799 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.435886 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.436329 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.436560 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:07:23.436884 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.429790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47599 (local address 127.0.97.1:46585)
0927 15:07:23.430007 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.430016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.430985 (+ 969us) server_negotiation.cc:407] Connection header received
0927 15:07:23.431203 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.431207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.431307 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.431405 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:23.432007 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.432734 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.434941 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.435907 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.436057 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.436156 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.436497 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.436655 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:07:23.436973 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.429862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46585 (local address 127.0.97.2:47599)
0927 15:07:23.430846 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.430867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.431035 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.431444 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.431449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.431491 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:23.431826 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.431835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.432901 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.432905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.434736 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.434750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.436129 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.436133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.436304 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.436337 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.436447 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.436594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":60,"thread_start_us":557,"threads_started":1}
I20260927 15:07:23.436904 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.429862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37971 (local address 127.0.97.2:58531)
0927 15:07:23.430731 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.430753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.430908 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.431321 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.431326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.431354 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:23.431811 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.431820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.432849 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.432854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.434475 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.434488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.435932 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.435938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.436128 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.436174 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.436271 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.436381 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":2,"thread_start_us":448,"threads_started":1}
I20260927 15:07:23.438025 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85a01406e2946a790a1aceaa3c9c18e" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878" is_pre_election: true
I20260927 15:07:23.438123 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85a01406e2946a790a1aceaa3c9c18e" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310" is_pre_election: true
I20260927 15:07:23.438594 8144 raft_consensus.cc:2454] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 0.
I20260927 15:07:23.438638 8428 raft_consensus.cc:2454] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 0.
I20260927 15:07:23.439397 8214 leader_election.cc:302] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1913adf0f67c403e9667952d11624878, 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:23.439841 8504 raft_consensus.cc:2781] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:23.440008 8504 raft_consensus.cc:482] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:23.440114 8504 raft_consensus.cc:3037] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:23.442428 8504 raft_consensus.cc:504] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.443132 8504 leader_election.cc:288] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 election: Requested vote from peers cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971), 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585)
I20260927 15:07:23.443720 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85a01406e2946a790a1aceaa3c9c18e" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310"
I20260927 15:07:23.443717 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85a01406e2946a790a1aceaa3c9c18e" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878"
I20260927 15:07:23.443950 8144 raft_consensus.cc:3037] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:23.443933 8428 raft_consensus.cc:3037] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:23.446226 8144 raft_consensus.cc:2454] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 1.
I20260927 15:07:23.446241 8428 raft_consensus.cc:2454] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 1.
I20260927 15:07:23.446774 8214 leader_election.cc:302] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:23.447196 8504 raft_consensus.cc:2781] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:23.447722 8504 raft_consensus.cc:686] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 1 LEADER]: Becoming Leader. State: Replica: 6336946210c5435588a37d69d3160037, State: Running, Role: LEADER
I20260927 15:07:23.448238 8504 consensus_queue.cc:237] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } }
I20260927 15:07:23.453572 7986 catalog_manager.cc:5495] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6336946210c5435588a37d69d3160037 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:23.477350 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.471447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.0.97.2:46245)
0927 15:07:23.471676 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.471688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.471814 (+ 126us) server_negotiation.cc:407] Connection header received
0927 15:07:23.472117 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.472121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.472241 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.472457 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:23.472852 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.473566 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.474993 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.475496 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.475633 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.476112 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:23.476693 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:23.476897 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.476970 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.477059 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:07:23.486590 8428 raft_consensus.cc:1261] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Refusing update from remote peer 6336946210c5435588a37d69d3160037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:23.486590 8144 raft_consensus.cc:1261] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Refusing update from remote peer 6336946210c5435588a37d69d3160037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:23.487691 8508 consensus_queue.cc:1035] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:23.488098 8504 consensus_queue.cc:1035] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:07:23.497538 8195 tablet.cc:2369] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:23.499439 8511 mvcc.cc:205] Tried to move back new op lower bound from 7333976651710967808 to 7333976651568545792. Current Snapshot: MvccSnapshot[applied={T|T < 7333976651710967808}]
I20260927 15:07:23.509166 8513 mvcc.cc:205] Tried to move back new op lower bound from 7333976651710967808 to 7333976651568545792. Current Snapshot: MvccSnapshot[applied={T|T < 7333976651710967808}]
I20260927 15:07:23.515516 7986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:23.515810 7986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:23.520385 7986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Sending DeleteTablet for 3 replicas of tablet d85a01406e2946a790a1aceaa3c9c18e
I20260927 15:07:23.521631 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet d85a01406e2946a790a1aceaa3c9c18e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49096
I20260927 15:07:23.521613 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet d85a01406e2946a790a1aceaa3c9c18e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58452
I20260927 15:07:23.522394 8124 tablet_service.cc:1514] Processing DeleteTablet for tablet d85a01406e2946a790a1aceaa3c9c18e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260927 15:07:23.524816 7986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:23.525785 7986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:23.529059 8522 tablet_replica.cc:330] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: stopping tablet replica
I20260927 15:07:23.529734 8522 raft_consensus.cc:2229] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:23.530397 8522 raft_consensus.cc:2258] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:23.531544 8521 tablet_replica.cc:330] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: stopping tablet replica
I20260927 15:07:23.532068 8521 raft_consensus.cc:2229] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:23.532761 8521 raft_consensus.cc:2258] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:23.533949 8520 tablet_replica.cc:330] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: stopping tablet replica
I20260927 15:07:23.534364 8520 raft_consensus.cc:2229] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:23.534865 8520 raft_consensus.cc:2258] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:23.536356 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd (DEFAULT_TABLE table=test-table [id=c17a2e14634a4d21a0dc5fd5d9fb660b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:23.536651 8124 tablet_service.cc:1467] Processing CreateTablet for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd (DEFAULT_TABLE table=test-table [id=c17a2e14634a4d21a0dc5fd5d9fb660b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:23.537012 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.537127 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.538285 8521 ts_tablet_manager.cc:1905] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:23.543572 8263 tablet_service.cc:1467] Processing CreateTablet for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd (DEFAULT_TABLE table=test-table [id=c17a2e14634a4d21a0dc5fd5d9fb660b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:23.543938 8498 tablet_bootstrap.cc:492] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Bootstrap starting.
I20260927 15:07:23.544121 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.545697 8498 tablet_bootstrap.cc:654] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.546756 8521 ts_tablet_manager.cc:1918] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:23.546988 8521 log.cc:1196] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/d85a01406e2946a790a1aceaa3c9c18e
I20260927 15:07:23.547874 8521 ts_tablet_manager.cc:1939] T d85a01406e2946a790a1aceaa3c9c18e P 6336946210c5435588a37d69d3160037: Deleting consensus metadata
I20260927 15:07:23.550479 7974 catalog_manager.cc:4841] TS 6336946210c5435588a37d69d3160037 (127.0.97.2:46245): tablet d85a01406e2946a790a1aceaa3c9c18e (table test-table [id=ffca7b2823204ee29de72eaa7d2a77ee]) successfully deleted
I20260927 15:07:23.552902 8498 tablet_bootstrap.cc:492] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: No bootstrap required, opened a new log
I20260927 15:07:23.553148 8498 ts_tablet_manager.cc:1397] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 15:07:23.553874 8498 raft_consensus.cc:348] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.554080 8498 raft_consensus.cc:374] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.554169 8498 raft_consensus.cc:729] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6b150470e746e6801fb8cdddfdf310, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.554456 8498 consensus_queue.cc:260] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.555172 8498 ts_tablet_manager.cc:1428] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:07:23.559401 8522 ts_tablet_manager.cc:1905] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:23.565081 8520 ts_tablet_manager.cc:1905] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:23.569917 8522 ts_tablet_manager.cc:1918] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:23.571848 8522 log.cc:1196] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/d85a01406e2946a790a1aceaa3c9c18e
I20260927 15:07:23.572481 8522 ts_tablet_manager.cc:1939] T d85a01406e2946a790a1aceaa3c9c18e P 1913adf0f67c403e9667952d11624878: Deleting consensus metadata
I20260927 15:07:23.572664 8520 ts_tablet_manager.cc:1918] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:23.572875 8520 log.cc:1196] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/d85a01406e2946a790a1aceaa3c9c18e
I20260927 15:07:23.573367 8520 ts_tablet_manager.cc:1939] T d85a01406e2946a790a1aceaa3c9c18e P cb6b150470e746e6801fb8cdddfdf310: Deleting consensus metadata
I20260927 15:07:23.574602 7973 catalog_manager.cc:4841] TS 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585): tablet d85a01406e2946a790a1aceaa3c9c18e (table test-table [id=ffca7b2823204ee29de72eaa7d2a77ee]) successfully deleted
I20260927 15:07:23.575248 7971 catalog_manager.cc:4841] TS cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971): tablet d85a01406e2946a790a1aceaa3c9c18e (table test-table [id=ffca7b2823204ee29de72eaa7d2a77ee]) successfully deleted
I20260927 15:07:23.575536 8499 tablet_bootstrap.cc:492] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Bootstrap starting.
I20260927 15:07:23.577386 8499 tablet_bootstrap.cc:654] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.580052 8497 tablet_bootstrap.cc:492] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Bootstrap starting.
I20260927 15:07:23.580354 8499 tablet_bootstrap.cc:492] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: No bootstrap required, opened a new log
I20260927 15:07:23.580508 8499 ts_tablet_manager.cc:1397] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:07:23.581410 8499 raft_consensus.cc:348] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.581648 8499 raft_consensus.cc:374] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.581763 8499 raft_consensus.cc:729] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1913adf0f67c403e9667952d11624878, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.582031 8497 tablet_bootstrap.cc:654] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.583750 8499 consensus_queue.cc:260] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.588027 8499 ts_tablet_manager.cc:1428] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260927 15:07:23.593631 8497 tablet_bootstrap.cc:492] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: No bootstrap required, opened a new log
I20260927 15:07:23.593858 8497 ts_tablet_manager.cc:1397] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 15:07:23.594659 8497 raft_consensus.cc:348] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.594942 8497 raft_consensus.cc:374] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.595062 8497 raft_consensus.cc:729] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6336946210c5435588a37d69d3160037, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.595468 8497 consensus_queue.cc:260] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.596264 8497 ts_tablet_manager.cc:1428] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:23.621510 8508 raft_consensus.cc:482] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:23.621788 8508 raft_consensus.cc:504] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.622597 8508 leader_election.cc:288] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:23.623304 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1fe55d40a4bb3b4c5b8e6080babcd" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310" is_pre_election: true
I20260927 15:07:23.623291 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1fe55d40a4bb3b4c5b8e6080babcd" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878" is_pre_election: true
I20260927 15:07:23.623530 8428 raft_consensus.cc:2454] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 0.
I20260927 15:07:23.623548 8144 raft_consensus.cc:2454] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 0.
I20260927 15:07:23.624099 8214 leader_election.cc:302] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1913adf0f67c403e9667952d11624878, 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:23.624497 8508 raft_consensus.cc:2781] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:23.624619 8508 raft_consensus.cc:482] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:23.624756 8508 raft_consensus.cc:3037] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:23.628407 8508 raft_consensus.cc:504] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.629258 8508 leader_election.cc:288] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 election: Requested vote from peers 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:23.629961 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1fe55d40a4bb3b4c5b8e6080babcd" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310"
I20260927 15:07:23.630023 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e1fe55d40a4bb3b4c5b8e6080babcd" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878"
I20260927 15:07:23.630239 8428 raft_consensus.cc:3037] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:23.630293 8144 raft_consensus.cc:3037] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:23.632555 8428 raft_consensus.cc:2454] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 1.
I20260927 15:07:23.633270 8214 leader_election.cc:302] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:23.633682 8508 raft_consensus.cc:2781] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:23.633759 8144 raft_consensus.cc:2454] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 1.
I20260927 15:07:23.634330 8508 raft_consensus.cc:686] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 1 LEADER]: Becoming Leader. State: Replica: 6336946210c5435588a37d69d3160037, State: Running, Role: LEADER
I20260927 15:07:23.634788 8508 consensus_queue.cc:237] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.637984 7987 catalog_manager.cc:5495] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6336946210c5435588a37d69d3160037 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:07:23 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:23.704416 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.681976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43843 (local address 127.0.0.1:46360)
0927 15:07:23.682719 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.682736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.682868 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.683324 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.683327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.684009 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:23.684277 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.684285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.685768 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.685772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.686818 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.686831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.687918 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.687927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.688112 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.695358 (+ 7246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:23.695382 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:23.697801 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:23.701898 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:23.701915 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:23.701922 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:23.702386 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:23.702813 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:23.702816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:23.702818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:23.702954 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:23.703581 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:23.703588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:23.703746 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.703879 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.704088 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":428,"threads_started":1}
I20260927 15:07:23.704809 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.681976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.0.97.62:43843)
0927 15:07:23.682235 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.682242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.682830 (+ 588us) server_negotiation.cc:407] Connection header received
0927 15:07:23.682991 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.682998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.683162 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.683279 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.684447 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.685586 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.686991 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.687701 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.687844 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.698073 (+ 10229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:23.698082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:23.698088 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:23.698132 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:23.701674 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.702519 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.702525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.702528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.702682 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.703097 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.703102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.703106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.703447 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:23.703568 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.704023 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.704258 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 15:07:23.710512 7987 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46360:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:07:23.711828 7987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:23.719116 8124 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8cf09518c44eda47b2c91e0b49803 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f0fe6d7e2a6433f98b49ae166f215e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:23.719425 8263 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8cf09518c44eda47b2c91e0b49803 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f0fe6d7e2a6433f98b49ae166f215e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:23.719782 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a8cf09518c44eda47b2c91e0b49803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.720033 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a8cf09518c44eda47b2c91e0b49803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.724090 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 42a8cf09518c44eda47b2c91e0b49803 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f0fe6d7e2a6433f98b49ae166f215e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:23.724705 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a8cf09518c44eda47b2c91e0b49803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:23.726483 8497 tablet_bootstrap.cc:492] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: Bootstrap starting.
I20260927 15:07:23.728447 8497 tablet_bootstrap.cc:654] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.729732 8499 tablet_bootstrap.cc:492] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: Bootstrap starting.
I20260927 15:07:23.731550 8497 tablet_bootstrap.cc:492] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: No bootstrap required, opened a new log
I20260927 15:07:23.731967 8497 ts_tablet_manager.cc:1397] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260927 15:07:23.732115 8499 tablet_bootstrap.cc:654] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.733129 8497 raft_consensus.cc:348] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.733601 8497 raft_consensus.cc:374] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.733788 8497 raft_consensus.cc:729] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6336946210c5435588a37d69d3160037, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.734295 8497 consensus_queue.cc:260] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.734997 8508 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:23.735342 8498 tablet_bootstrap.cc:492] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: Bootstrap starting.
I20260927 15:07:23.735363 8508 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:23.735220 8497 ts_tablet_manager.cc:1428] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:07:23.737064 8498 tablet_bootstrap.cc:654] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:23.744771 8499 tablet_bootstrap.cc:492] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: No bootstrap required, opened a new log
I20260927 15:07:23.744822 8498 tablet_bootstrap.cc:492] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: No bootstrap required, opened a new log
I20260927 15:07:23.744984 8498 ts_tablet_manager.cc:1397] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260927 15:07:23.744997 8499 ts_tablet_manager.cc:1397] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 15:07:23.745657 8499 raft_consensus.cc:348] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.745733 8498 raft_consensus.cc:348] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.745909 8499 raft_consensus.cc:374] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.745997 8498 raft_consensus.cc:374] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:23.746078 8498 raft_consensus.cc:729] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6b150470e746e6801fb8cdddfdf310, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.745996 8499 raft_consensus.cc:729] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1913adf0f67c403e9667952d11624878, State: Initialized, Role: FOLLOWER
I20260927 15:07:23.746372 8498 consensus_queue.cc:260] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.746815 8503 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:23.747129 8503 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:23.747417 8499 consensus_queue.cc:260] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.747946 8505 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:23.748342 8505 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:23.748795 8498 ts_tablet_manager.cc:1428] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 15:07:23.750789 8499 ts_tablet_manager.cc:1428] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
W20260927 15:07:23.793359 8334 tablet.cc:2369] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:23.988811 8505 raft_consensus.cc:482] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:23.989063 8505 raft_consensus.cc:504] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:23.990825 8505 leader_election.cc:288] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6336946210c5435588a37d69d3160037 (127.0.97.2:46245), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:24.018325 8508 raft_consensus.cc:482] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:24.018577 8508 raft_consensus.cc:504] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.019467 8508 leader_election.cc:288] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:24.020476 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310" is_pre_election: true
I20260927 15:07:24.020787 8428 raft_consensus.cc:2454] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 0.
I20260927 15:07:24.021587 8214 leader_election.cc:302] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.022017 8508 raft_consensus.cc:2781] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:24.022145 8508 raft_consensus.cc:482] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:24.022257 8508 raft_consensus.cc:3037] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.025640 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.991350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37971 (local address 127.0.97.1:46323)
0927 15:07:24.017575 (+ 26225us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.017598 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.017779 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.018592 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.018597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.018625 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:24.018996 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.019010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.020600 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.020606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.022699 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.022721 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.024712 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.024727 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.024949 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.024984 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.025095 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.025245 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26101,"thread_start_us":26568,"threads_started":1}
I20260927 15:07:24.025928 8508 raft_consensus.cc:504] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.026394 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.991824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46323 (local address 127.0.97.3:37971)
0927 15:07:24.017407 (+ 25583us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.017415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.017893 (+ 478us) server_negotiation.cc:407] Connection header received
0927 15:07:24.018198 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.018204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.018286 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.018391 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:24.019212 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.020402 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.022913 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.024167 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.024458 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.024607 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.025955 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.026117 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25438,"thread_start_us":25322,"threads_started":1}
I20260927 15:07:24.026934 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310" is_pre_election: true
I20260927 15:07:24.027216 8428 raft_consensus.cc:2454] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1913adf0f67c403e9667952d11624878 in term 0.
I20260927 15:07:24.027791 8074 leader_election.cc:302] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1913adf0f67c403e9667952d11624878, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.028031 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310"
I20260927 15:07:24.028203 8505 raft_consensus.cc:2781] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:24.028311 8505 raft_consensus.cc:482] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:24.028345 8428 raft_consensus.cc:3037] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.028386 8505 raft_consensus.cc:3037] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.031633 8505 raft_consensus.cc:504] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.032251 8428 raft_consensus.cc:2454] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6336946210c5435588a37d69d3160037 in term 1.
I20260927 15:07:24.032461 8505 leader_election.cc:288] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 election: Requested vote from peers 6336946210c5435588a37d69d3160037 (127.0.97.2:46245), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:24.032929 8214 leader_election.cc:302] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.033308 8504 raft_consensus.cc:2781] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:24.033201 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310"
I20260927 15:07:24.033543 8427 raft_consensus.cc:2379] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1913adf0f67c403e9667952d11624878 in current term 1: Already voted for candidate 6336946210c5435588a37d69d3160037 in this term.
I20260927 15:07:24.033749 8504 raft_consensus.cc:686] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [term 1 LEADER]: Becoming Leader. State: Replica: 6336946210c5435588a37d69d3160037, State: Running, Role: LEADER
I20260927 15:07:24.034067 8504 consensus_queue.cc:237] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.035852 8508 leader_election.cc:288] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [CANDIDATE]: Term 1 election: Requested vote from peers 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:24.036583 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878" is_pre_election: true
I20260927 15:07:24.036931 8144 raft_consensus.cc:2379] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6336946210c5435588a37d69d3160037 in current term 1: Already voted for candidate 1913adf0f67c403e9667952d11624878 in this term.
I20260927 15:07:24.037600 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "6336946210c5435588a37d69d3160037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878"
I20260927 15:07:24.037921 8143 raft_consensus.cc:2379] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6336946210c5435588a37d69d3160037 in current term 1: Already voted for candidate 1913adf0f67c403e9667952d11624878 in this term.
I20260927 15:07:24.045535 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.991656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46245 (local address 127.0.97.1:50993)
0927 15:07:24.017974 (+ 26318us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.017997 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.036792 (+ 18795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.038395 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.038400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.038429 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:24.038778 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.038789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.040254 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.040265 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.042416 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.042432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.044685 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.044693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.044893 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.044925 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.045021 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.045180 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26178,"thread_start_us":25758,"threads_started":1}
I20260927 15:07:24.046046 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.991351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50993 (local address 127.0.97.2:46245)
0927 15:07:24.016817 (+ 25466us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.016827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.036667 (+ 19840us) server_negotiation.cc:407] Connection header received
0927 15:07:24.036987 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.036992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.037069 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.037189 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:24.038962 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.040075 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.042626 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.044259 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.044422 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.044537 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.045310 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.045453 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25284,"thread_start_us":44640,"threads_started":1}
I20260927 15:07:24.046965 8282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6336946210c5435588a37d69d3160037"
I20260927 15:07:24.046977 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a8cf09518c44eda47b2c91e0b49803" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6336946210c5435588a37d69d3160037" is_pre_election: true
I20260927 15:07:24.048418 8077 leader_election.cc:302] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1913adf0f67c403e9667952d11624878; no voters: 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310
I20260927 15:07:24.048774 8505 raft_consensus.cc:2726] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 15:07:24.051980 8540 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: TxnStatusTablet state changed. Reason: New leader 6336946210c5435588a37d69d3160037. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.052683 8540 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:24.053504 8542 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:07:24.053889 8542 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:07:24.054255 7987 catalog_manager.cc:5495] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6336946210c5435588a37d69d3160037 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:24.118744 8540 consensus_queue.cc:1035] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.145833 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.133044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46245 (local address 127.0.0.1:49118)
0927 15:07:24.133342 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.133369 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.133599 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.135625 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.135634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.135655 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:24.135966 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.135982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.139316 (+ 3334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.139324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.141148 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.141167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.143164 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.143174 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.143475 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.143592 (+ 117us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.144938 (+ 1346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.144973 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.145051 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.145292 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 15:07:24.147248 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.132820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.0.97.2:46245)
0927 15:07:24.134599 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.134609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.134635 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:07:24.134731 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.134739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.134855 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.136166 (+ 1311us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:24.136350 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.137578 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.141401 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.142076 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.143884 (+ 1808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.144044 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.144227 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.146637 (+ 2410us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.146724 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.146851 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576}
I20260927 15:07:24.151250 8508 consensus_queue.cc:1035] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.167894 8428 raft_consensus.cc:1261] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Refusing update from remote peer 6336946210c5435588a37d69d3160037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:24.168700 8508 consensus_queue.cc:1035] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.171638 8143 raft_consensus.cc:1261] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Refusing update from remote peer 6336946210c5435588a37d69d3160037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:24.172534 8503 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: TxnStatusTablet state changed. Reason: New leader 6336946210c5435588a37d69d3160037. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.172842 8503 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:24.173669 8508 consensus_queue.cc:1035] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.183419 8505 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: TxnStatusTablet state changed. Reason: New leader 6336946210c5435588a37d69d3160037. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.183881 8505 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:24.191977 8503 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.192320 8503 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P cb6b150470e746e6801fb8cdddfdf310: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:24.196350 8540 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.196656 8540 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:24.198712 8504 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.199060 8504 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 6336946210c5435588a37d69d3160037: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:24.200065 8505 tablet_replica.cc:439] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6336946210c5435588a37d69d3160037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } } }
I20260927 15:07:24.200347 8505 tablet_replica.cc:442] T 42a8cf09518c44eda47b2c91e0b49803 P 1913adf0f67c403e9667952d11624878: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:24.233078 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.225353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44131 (local address 127.0.97.2:46245)
0927 15:07:24.225531 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.225538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.226171 (+ 633us) server_negotiation.cc:407] Connection header received
0927 15:07:24.226425 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.226430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.226502 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.226557 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:24.227337 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.228575 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.230787 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.231523 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.231624 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.232270 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.232425 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.232483 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.232670 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.232757 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:07:24.233943 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.225313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46245 (local address 127.0.97.2:44131)
0927 15:07:24.226058 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.226078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.226287 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.226735 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.226739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.226750 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:24.227169 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.227183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.228735 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.228740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.230609 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.230627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.231815 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.231827 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.231985 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.232062 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.232544 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.232552 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.233391 (+ 839us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.233548 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":364,"threads_started":1}
I20260927 15:07:24.326515 7987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:24.326774 7987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:24.330655 7987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Sending DeleteTablet for 3 replicas of tablet 60e1fe55d40a4bb3b4c5b8e6080babcd
I20260927 15:07:24.331616 8124 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260927 15:07:24.332298 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58452
I20260927 15:07:24.334739 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e1fe55d40a4bb3b4c5b8e6080babcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49096
I20260927 15:07:24.348107 8559 tablet_replica.cc:330] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: stopping tablet replica
I20260927 15:07:24.350207 8558 tablet_replica.cc:330] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: stopping tablet replica
I20260927 15:07:24.350852 8560 tablet_replica.cc:330] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: stopping tablet replica
I20260927 15:07:24.351660 8558 raft_consensus.cc:2229] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:24.351902 8560 raft_consensus.cc:2229] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:24.352692 8558 raft_consensus.cc:2258] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.353063 8560 raft_consensus.cc:2258] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.354585 8558 ts_tablet_manager.cc:1905] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.356098 8560 ts_tablet_manager.cc:1905] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.359302 8559 raft_consensus.cc:2229] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:24.359750 8559 raft_consensus.cc:2258] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.361013 8559 ts_tablet_manager.cc:1905] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.365628 8560 ts_tablet_manager.cc:1918] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:24.369302 8559 ts_tablet_manager.cc:1918] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:24.369578 8559 log.cc:1196] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/60e1fe55d40a4bb3b4c5b8e6080babcd
I20260927 15:07:24.370204 8559 ts_tablet_manager.cc:1939] T 60e1fe55d40a4bb3b4c5b8e6080babcd P cb6b150470e746e6801fb8cdddfdf310: Deleting consensus metadata
I20260927 15:07:24.377743 7971 catalog_manager.cc:4841] TS cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971): tablet 60e1fe55d40a4bb3b4c5b8e6080babcd (table test-table [id=c17a2e14634a4d21a0dc5fd5d9fb660b]) successfully deleted
I20260927 15:07:24.378733 8558 ts_tablet_manager.cc:1918] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:24.378929 8558 log.cc:1196] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/60e1fe55d40a4bb3b4c5b8e6080babcd
I20260927 15:07:24.379740 8558 ts_tablet_manager.cc:1939] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 1913adf0f67c403e9667952d11624878: Deleting consensus metadata
I20260927 15:07:24.381424 7973 catalog_manager.cc:4841] TS 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585): tablet 60e1fe55d40a4bb3b4c5b8e6080babcd (table test-table [id=c17a2e14634a4d21a0dc5fd5d9fb660b]) successfully deleted
I20260927 15:07:24.381834 8560 log.cc:1196] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/60e1fe55d40a4bb3b4c5b8e6080babcd
I20260927 15:07:24.382707 8560 ts_tablet_manager.cc:1939] T 60e1fe55d40a4bb3b4c5b8e6080babcd P 6336946210c5435588a37d69d3160037: Deleting consensus metadata
Sep 27 15:07:24 dist-test-slave-g0w1 krb5kdc[7942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260927 15:07:24.390616 7974 catalog_manager.cc:4841] TS 6336946210c5435588a37d69d3160037 (127.0.97.2:46245): tablet 60e1fe55d40a4bb3b4c5b8e6080babcd (table test-table [id=c17a2e14634a4d21a0dc5fd5d9fb660b]) successfully deleted
Sep 27 15:07:24 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:24 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:24.438117 8568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:24.423249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.0.97.62:43843)
0927 15:07:24.424009 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.424016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.424038 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:07:24.424102 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.424108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.424184 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.424309 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.426355 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.427523 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.429557 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.430207 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.430346 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.437724 (+ 7378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46366: BlockingRecv error: recv got EOF from 127.0.0.1:46366 (error 108)
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":400,"threads_started":1}
Sep 27 15:07:24 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 15:07:24.464633 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.442184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.97.62:43843)
0927 15:07:24.442375 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.442381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.442934 (+ 553us) server_negotiation.cc:407] Connection header received
0927 15:07:24.443200 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.443209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.443325 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.443456 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.445177 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.446706 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.448747 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.449712 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.449902 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.455711 (+ 5809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.455723 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.455731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.455801 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.460770 (+ 4969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.462030 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.462039 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.462045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.462271 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.462827 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.462836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.462841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.463350 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.463620 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.463991 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.464248 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:07:24.468817 7987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:24.469513 7987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:24.477954 8124 tablet_service.cc:1467] Processing CreateTablet for tablet 617a5a41f5934ae38a4e627ab6ceb64e (DEFAULT_TABLE table=test-table [id=185912cf18114737988db69f00c42d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:24.477963 8263 tablet_service.cc:1467] Processing CreateTablet for tablet 617a5a41f5934ae38a4e627ab6ceb64e (DEFAULT_TABLE table=test-table [id=185912cf18114737988db69f00c42d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:24.478428 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617a5a41f5934ae38a4e627ab6ceb64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.478434 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617a5a41f5934ae38a4e627ab6ceb64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.481518 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 617a5a41f5934ae38a4e627ab6ceb64e (DEFAULT_TABLE table=test-table [id=185912cf18114737988db69f00c42d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:24.481982 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617a5a41f5934ae38a4e627ab6ceb64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.485597 8574 tablet_bootstrap.cc:492] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Bootstrap starting.
I20260927 15:07:24.488076 8574 tablet_bootstrap.cc:654] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.490412 8574 tablet_bootstrap.cc:492] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: No bootstrap required, opened a new log
I20260927 15:07:24.490583 8574 ts_tablet_manager.cc:1397] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:07:24.491422 8574 raft_consensus.cc:348] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.491691 8574 raft_consensus.cc:374] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.491782 8574 raft_consensus.cc:729] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1913adf0f67c403e9667952d11624878, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.492063 8574 consensus_queue.cc:260] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.492645 8574 ts_tablet_manager.cc:1428] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:24.489693 8575 tablet_bootstrap.cc:492] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Bootstrap starting.
I20260927 15:07:24.494604 8575 tablet_bootstrap.cc:654] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.496304 8577 tablet_bootstrap.cc:492] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Bootstrap starting.
I20260927 15:07:24.498354 8577 tablet_bootstrap.cc:654] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.500545 8575 tablet_bootstrap.cc:492] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: No bootstrap required, opened a new log
I20260927 15:07:24.500730 8575 ts_tablet_manager.cc:1397] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260927 15:07:24.501540 8575 raft_consensus.cc:348] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.501820 8575 raft_consensus.cc:374] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.501940 8575 raft_consensus.cc:729] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6b150470e746e6801fb8cdddfdf310, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.502210 8575 consensus_queue.cc:260] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.502866 8575 ts_tablet_manager.cc:1428] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:24.508895 8577 tablet_bootstrap.cc:492] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: No bootstrap required, opened a new log
I20260927 15:07:24.509114 8577 ts_tablet_manager.cc:1397] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260927 15:07:24.510027 8577 raft_consensus.cc:348] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.510316 8577 raft_consensus.cc:374] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.510422 8577 raft_consensus.cc:729] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6336946210c5435588a37d69d3160037, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.510704 8577 consensus_queue.cc:260] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.511426 8577 ts_tablet_manager.cc:1428] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:24.559569 8505 raft_consensus.cc:482] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:24.559804 8505 raft_consensus.cc:504] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.560534 8505 leader_election.cc:288] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6336946210c5435588a37d69d3160037 (127.0.97.2:46245), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:24.561187 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617a5a41f5934ae38a4e627ab6ceb64e" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6336946210c5435588a37d69d3160037" is_pre_election: true
I20260927 15:07:24.561235 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617a5a41f5934ae38a4e627ab6ceb64e" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310" is_pre_election: true
I20260927 15:07:24.561461 8427 raft_consensus.cc:2454] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1913adf0f67c403e9667952d11624878 in term 0.
I20260927 15:07:24.561517 8283 raft_consensus.cc:2454] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1913adf0f67c403e9667952d11624878 in term 0.
I20260927 15:07:24.561959 8074 leader_election.cc:302] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1913adf0f67c403e9667952d11624878, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.562394 8505 raft_consensus.cc:2781] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:24.562510 8505 raft_consensus.cc:482] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:24.562605 8505 raft_consensus.cc:3037] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.565505 8505 raft_consensus.cc:504] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.566663 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617a5a41f5934ae38a4e627ab6ceb64e" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6336946210c5435588a37d69d3160037"
I20260927 15:07:24.566701 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617a5a41f5934ae38a4e627ab6ceb64e" candidate_uuid: "1913adf0f67c403e9667952d11624878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6b150470e746e6801fb8cdddfdf310"
I20260927 15:07:24.566901 8427 raft_consensus.cc:3037] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.566923 8283 raft_consensus.cc:3037] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.568959 8505 leader_election.cc:288] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 election: Requested vote from peers 6336946210c5435588a37d69d3160037 (127.0.97.2:46245), cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971)
I20260927 15:07:24.569783 8427 raft_consensus.cc:2454] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1913adf0f67c403e9667952d11624878 in term 1.
I20260927 15:07:24.570322 8074 leader_election.cc:302] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1913adf0f67c403e9667952d11624878, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.570554 8283 raft_consensus.cc:2454] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1913adf0f67c403e9667952d11624878 in term 1.
I20260927 15:07:24.570657 8505 raft_consensus.cc:2781] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:24.571277 8505 raft_consensus.cc:686] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 1 LEADER]: Becoming Leader. State: Replica: 1913adf0f67c403e9667952d11624878, State: Running, Role: LEADER
I20260927 15:07:24.571712 8505 consensus_queue.cc:237] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.575812 7986 catalog_manager.cc:5495] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1913adf0f67c403e9667952d11624878 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1913adf0f67c403e9667952d11624878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:24.596647 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.589643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.0.97.1:46585)
0927 15:07:24.590266 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.590273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.590297 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:07:24.590402 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.590408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.590502 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.590683 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:24.591055 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.592234 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.593634 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.594325 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.594449 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.594942 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.595890 (+ 948us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.596046 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.596225 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.596322 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":332,"threads_started":1}
I20260927 15:07:24.600998 8427 raft_consensus.cc:1261] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Refusing update from remote peer 1913adf0f67c403e9667952d11624878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:24.601634 8283 raft_consensus.cc:1261] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Refusing update from remote peer 1913adf0f67c403e9667952d11624878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:24.601795 8580 consensus_queue.cc:1035] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.602432 8505 consensus_queue.cc:1035] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.607125 8548 mvcc.cc:205] Tried to move back new op lower bound from 7333976656277618688 to 7333976656170233856. Current Snapshot: MvccSnapshot[applied={T|T < 7333976656277618688}]
I20260927 15:07:24.620827 7986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:24.621086 7986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:24.624472 7986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Sending DeleteTablet for 3 replicas of tablet 617a5a41f5934ae38a4e627ab6ceb64e
I20260927 15:07:24.625413 8124 tablet_service.cc:1514] Processing DeleteTablet for tablet 617a5a41f5934ae38a4e627ab6ceb64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260927 15:07:24.625595 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet 617a5a41f5934ae38a4e627ab6ceb64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49096
I20260927 15:07:24.625733 8558 tablet_replica.cc:330] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: stopping tablet replica
I20260927 15:07:24.625883 8560 tablet_replica.cc:330] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: stopping tablet replica
I20260927 15:07:24.625937 8558 raft_consensus.cc:2229] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:24.626075 8560 raft_consensus.cc:2229] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:24.626327 8560 raft_consensus.cc:2258] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.626500 8558 raft_consensus.cc:2258] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.626703 7986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:07:24.627056 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 617a5a41f5934ae38a4e627ab6ceb64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58452
I20260927 15:07:24.627369 8559 tablet_replica.cc:330] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: stopping tablet replica
W20260927 15:07:24.627496 7986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:24.627565 8559 raft_consensus.cc:2229] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:24.627774 8559 raft_consensus.cc:2258] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.627959 8558 ts_tablet_manager.cc:1905] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.629155 8560 ts_tablet_manager.cc:1905] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.631346 8559 ts_tablet_manager.cc:1905] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.635921 8558 ts_tablet_manager.cc:1918] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:24.636089 8558 log.cc:1196] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/617a5a41f5934ae38a4e627ab6ceb64e
I20260927 15:07:24.636168 8560 ts_tablet_manager.cc:1918] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:24.636332 8560 log.cc:1196] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/617a5a41f5934ae38a4e627ab6ceb64e
I20260927 15:07:24.636549 8558 ts_tablet_manager.cc:1939] T 617a5a41f5934ae38a4e627ab6ceb64e P 1913adf0f67c403e9667952d11624878: Deleting consensus metadata
I20260927 15:07:24.636763 8560 ts_tablet_manager.cc:1939] T 617a5a41f5934ae38a4e627ab6ceb64e P 6336946210c5435588a37d69d3160037: Deleting consensus metadata
I20260927 15:07:24.637826 7973 catalog_manager.cc:4841] TS 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585): tablet 617a5a41f5934ae38a4e627ab6ceb64e (table test-table [id=185912cf18114737988db69f00c42d7d]) successfully deleted
I20260927 15:07:24.637929 7974 catalog_manager.cc:4841] TS 6336946210c5435588a37d69d3160037 (127.0.97.2:46245): tablet 617a5a41f5934ae38a4e627ab6ceb64e (table test-table [id=185912cf18114737988db69f00c42d7d]) successfully deleted
I20260927 15:07:24.637992 8559 ts_tablet_manager.cc:1918] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:24.638211 8559 log.cc:1196] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/617a5a41f5934ae38a4e627ab6ceb64e
I20260927 15:07:24.638687 8559 ts_tablet_manager.cc:1939] T 617a5a41f5934ae38a4e627ab6ceb64e P cb6b150470e746e6801fb8cdddfdf310: Deleting consensus metadata
I20260927 15:07:24.639885 7971 catalog_manager.cc:4841] TS cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971): tablet 617a5a41f5934ae38a4e627ab6ceb64e (table test-table [id=185912cf18114737988db69f00c42d7d]) successfully deleted
I20260927 15:07:24.642956 8124 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea63e471e504999a7c85ef322e6ede0 (DEFAULT_TABLE table=test-table [id=c9a3e9d853a54cd3bbb09dca318fa84b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:24.642976 8263 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea63e471e504999a7c85ef322e6ede0 (DEFAULT_TABLE table=test-table [id=c9a3e9d853a54cd3bbb09dca318fa84b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:24.643486 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea63e471e504999a7c85ef322e6ede0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.643496 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea63e471e504999a7c85ef322e6ede0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.647192 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea63e471e504999a7c85ef322e6ede0 (DEFAULT_TABLE table=test-table [id=c9a3e9d853a54cd3bbb09dca318fa84b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:24.647686 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea63e471e504999a7c85ef322e6ede0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.649353 8574 tablet_bootstrap.cc:492] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Bootstrap starting.
I20260927 15:07:24.651213 8574 tablet_bootstrap.cc:654] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.652714 8577 tablet_bootstrap.cc:492] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Bootstrap starting.
I20260927 15:07:24.654282 8575 tablet_bootstrap.cc:492] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Bootstrap starting.
I20260927 15:07:24.654394 8577 tablet_bootstrap.cc:654] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.655977 8575 tablet_bootstrap.cc:654] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.657430 8574 tablet_bootstrap.cc:492] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: No bootstrap required, opened a new log
I20260927 15:07:24.657707 8574 ts_tablet_manager.cc:1397] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260927 15:07:24.658876 8574 raft_consensus.cc:348] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.659369 8574 raft_consensus.cc:374] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.659512 8574 raft_consensus.cc:729] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1913adf0f67c403e9667952d11624878, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.659894 8574 consensus_queue.cc:260] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.660567 8574 ts_tablet_manager.cc:1428] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:24.669417 8575 tablet_bootstrap.cc:492] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: No bootstrap required, opened a new log
I20260927 15:07:24.669629 8575 ts_tablet_manager.cc:1397] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260927 15:07:24.671720 8575 raft_consensus.cc:348] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.672030 8575 raft_consensus.cc:374] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.672127 8575 raft_consensus.cc:729] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6b150470e746e6801fb8cdddfdf310, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.672396 8575 consensus_queue.cc:260] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.672798 8577 tablet_bootstrap.cc:492] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: No bootstrap required, opened a new log
I20260927 15:07:24.672981 8577 ts_tablet_manager.cc:1397] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20260927 15:07:24.673718 8575 ts_tablet_manager.cc:1428] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:24.673810 8577 raft_consensus.cc:348] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.674090 8577 raft_consensus.cc:374] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.674176 8577 raft_consensus.cc:729] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6336946210c5435588a37d69d3160037, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.674427 8577 consensus_queue.cc:260] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.675038 8577 ts_tablet_manager.cc:1428] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 15:07:24.739126 8503 raft_consensus.cc:482] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:24.739454 8503 raft_consensus.cc:504] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.741237 8503 leader_election.cc:288] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585), 6336946210c5435588a37d69d3160037 (127.0.97.2:46245)
I20260927 15:07:24.758932 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.742009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34649 (local address 127.0.97.1:46585)
0927 15:07:24.748098 (+ 6089us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.748106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.751184 (+ 3078us) server_negotiation.cc:407] Connection header received
0927 15:07:24.751869 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.751875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.751939 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.751997 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:24.753067 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.754132 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.756419 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.757792 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.757951 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.758067 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.758465 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.758629 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5905}
I20260927 15:07:24.759280 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.741456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46585 (local address 127.0.97.3:34649)
0927 15:07:24.750947 (+ 9491us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.750997 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.751655 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.752166 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.752169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.752189 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:24.752864 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.752881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.754303 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.754306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.756215 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.756235 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.758008 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.758015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.758218 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.758260 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.758407 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.758566 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9240,"mutex_wait_us":130,"thread_start_us":9043,"threads_started":1}
I20260927 15:07:24.759646 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.741690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47443 (local address 127.0.97.2:46245)
0927 15:07:24.751953 (+ 10263us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.751959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.751980 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:07:24.752050 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.752054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.752138 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.752254 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:24.752912 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.754034 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.756860 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.758060 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.758195 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.758307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.759167 (+ 860us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.759341 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10122,"thread_start_us":10006,"threads_started":1}
I20260927 15:07:24.759950 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.741492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46245 (local address 127.0.97.3:47443)
0927 15:07:24.751292 (+ 9800us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.751312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.751472 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.752295 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.752299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.752336 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:24.752731 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.752742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.754212 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.754218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.756665 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.756683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.758280 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.758289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.758484 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.758515 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.758604 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.758734 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9666,"mutex_wait_us":7,"thread_start_us":9738,"threads_started":1}
I20260927 15:07:24.760053 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea63e471e504999a7c85ef322e6ede0" candidate_uuid: "cb6b150470e746e6801fb8cdddfdf310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878" is_pre_election: true
I20260927 15:07:24.760414 8143 raft_consensus.cc:2454] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6b150470e746e6801fb8cdddfdf310 in term 0.
I20260927 15:07:24.760993 8355 leader_election.cc:302] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1913adf0f67c403e9667952d11624878, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.761152 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea63e471e504999a7c85ef322e6ede0" candidate_uuid: "cb6b150470e746e6801fb8cdddfdf310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6336946210c5435588a37d69d3160037" is_pre_election: true
I20260927 15:07:24.761459 8503 raft_consensus.cc:2781] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:24.761435 8283 raft_consensus.cc:2454] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6b150470e746e6801fb8cdddfdf310 in term 0.
I20260927 15:07:24.761572 8503 raft_consensus.cc:482] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:24.761658 8503 raft_consensus.cc:3037] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.764084 8503 raft_consensus.cc:504] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.764694 8503 leader_election.cc:288] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [CANDIDATE]: Term 1 election: Requested vote from peers 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585), 6336946210c5435588a37d69d3160037 (127.0.97.2:46245)
I20260927 15:07:24.765164 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea63e471e504999a7c85ef322e6ede0" candidate_uuid: "cb6b150470e746e6801fb8cdddfdf310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6336946210c5435588a37d69d3160037"
I20260927 15:07:24.765389 8283 raft_consensus.cc:3037] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.767350 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea63e471e504999a7c85ef322e6ede0" candidate_uuid: "cb6b150470e746e6801fb8cdddfdf310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1913adf0f67c403e9667952d11624878"
I20260927 15:07:24.767460 8283 raft_consensus.cc:2454] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6b150470e746e6801fb8cdddfdf310 in term 1.
I20260927 15:07:24.767612 8143 raft_consensus.cc:3037] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.767990 8356 leader_election.cc:302] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6336946210c5435588a37d69d3160037, cb6b150470e746e6801fb8cdddfdf310; no voters:
I20260927 15:07:24.768352 8503 raft_consensus.cc:2781] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:24.768900 8503 raft_consensus.cc:686] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 1 LEADER]: Becoming Leader. State: Replica: cb6b150470e746e6801fb8cdddfdf310, State: Running, Role: LEADER
I20260927 15:07:24.769788 8503 consensus_queue.cc:237] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } }
I20260927 15:07:24.770897 8143 raft_consensus.cc:2454] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6b150470e746e6801fb8cdddfdf310 in term 1.
I20260927 15:07:24.773517 7987 catalog_manager.cc:5495] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cb6b150470e746e6801fb8cdddfdf310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6b150470e746e6801fb8cdddfdf310" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37971 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:24.805838 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.799812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.0.97.3:37971)
0927 15:07:24.800614 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.800618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.800636 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:07:24.800719 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.800722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.800791 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.800972 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:24.801404 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.802170 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.803581 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.803993 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.804136 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.804596 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.805193 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.805362 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.805411 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.805490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":583,"threads_started":1}
I20260927 15:07:24.822428 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.816701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38967 (local address 127.0.97.2:46245)
0927 15:07:24.816907 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.816914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.817322 (+ 408us) server_negotiation.cc:407] Connection header received
0927 15:07:24.817558 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.817563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.817652 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.817790 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:24.818342 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.819228 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.820714 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.821100 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.821244 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.821671 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.821793 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.821887 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.822037 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.822115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:07:24.822532 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.816544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46245 (local address 127.0.97.3:38967)
0927 15:07:24.817214 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.817229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.817415 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.817802 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.817806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.817817 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:24.818133 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.818145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.819359 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.819365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.820554 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.820566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.821278 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.821286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.821420 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.821475 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.821908 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.821920 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.822014 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.822146 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":269,"threads_started":1}
I20260927 15:07:24.832710 8143 raft_consensus.cc:1261] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Refusing update from remote peer cb6b150470e746e6801fb8cdddfdf310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:24.832700 8283 raft_consensus.cc:1261] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Refusing update from remote peer cb6b150470e746e6801fb8cdddfdf310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:24.833467 8593 consensus_queue.cc:1035] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1913adf0f67c403e9667952d11624878" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.833896 8503 consensus_queue.cc:1035] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6336946210c5435588a37d69d3160037" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:24.874135 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.865244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55341 (local address 127.0.97.3:37971)
0927 15:07:24.865401 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.865409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.867721 (+ 2312us) server_negotiation.cc:407] Connection header received
0927 15:07:24.867864 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.867869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.867950 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.868045 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:24.868705 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.869542 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.871931 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.872618 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.872724 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.873337 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.873476 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.873563 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.873724 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.873804 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:24.874153 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.864906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37971 (local address 127.0.97.2:55341)
0927 15:07:24.867578 (+ 2672us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.867600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.867745 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.868227 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.868233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.868246 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:24.868532 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.868542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.869684 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.869693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.871282 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.871764 (+ 482us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.872933 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.872940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.873121 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.873180 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:24.873565 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:24.873580 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.873661 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.873810 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2505,"thread_start_us":2366,"threads_started":1}
I20260927 15:07:24.923209 7986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:24.923502 7986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:24.927222 7986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4babd43ee713449f8d4e774deeb47fad: Sending DeleteTablet for 3 replicas of tablet 3ea63e471e504999a7c85ef322e6ede0
I20260927 15:07:24.928613 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea63e471e504999a7c85ef322e6ede0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49096
I20260927 15:07:24.928943 8560 tablet_replica.cc:330] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: stopping tablet replica
I20260927 15:07:24.929155 8560 raft_consensus.cc:2229] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:24.929383 8560 raft_consensus.cc:2258] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.929555 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea63e471e504999a7c85ef322e6ede0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58452
I20260927 15:07:24.929908 8559 tablet_replica.cc:330] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: stopping tablet replica
I20260927 15:07:24.930114 8559 raft_consensus.cc:2229] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:24.930603 8560 ts_tablet_manager.cc:1905] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.930668 8559 raft_consensus.cc:2258] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.932054 8559 ts_tablet_manager.cc:1905] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.937592 8560 ts_tablet_manager.cc:1918] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:24.937758 8560 log.cc:1196] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/3ea63e471e504999a7c85ef322e6ede0
I20260927 15:07:24.938356 8560 ts_tablet_manager.cc:1939] T 3ea63e471e504999a7c85ef322e6ede0 P 6336946210c5435588a37d69d3160037: Deleting consensus metadata
I20260927 15:07:24.939643 7974 catalog_manager.cc:4841] TS 6336946210c5435588a37d69d3160037 (127.0.97.2:46245): tablet 3ea63e471e504999a7c85ef322e6ede0 (table test-table [id=c9a3e9d853a54cd3bbb09dca318fa84b]) successfully deleted
I20260927 15:07:24.940198 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 8058
I20260927 15:07:24.943758 8124 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea63e471e504999a7c85ef322e6ede0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260927 15:07:24.944132 8558 tablet_replica.cc:330] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: stopping tablet replica
I20260927 15:07:24.944347 8558 raft_consensus.cc:2229] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:24.944586 8558 raft_consensus.cc:2258] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:24.945943 8558 ts_tablet_manager.cc:1905] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:24.949124 8559 ts_tablet_manager.cc:1918] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:24.949373 8559 log.cc:1196] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/3ea63e471e504999a7c85ef322e6ede0
I20260927 15:07:24.949945 8559 ts_tablet_manager.cc:1939] T 3ea63e471e504999a7c85ef322e6ede0 P cb6b150470e746e6801fb8cdddfdf310: Deleting consensus metadata
I20260927 15:07:24.951297 7971 catalog_manager.cc:4841] TS cb6b150470e746e6801fb8cdddfdf310 (127.0.97.3:37971): tablet 3ea63e471e504999a7c85ef322e6ede0 (table test-table [id=c9a3e9d853a54cd3bbb09dca318fa84b]) successfully deleted
I20260927 15:07:24.953785 8558 ts_tablet_manager.cc:1918] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:24.953944 8558 log.cc:1196] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/3ea63e471e504999a7c85ef322e6ede0
I20260927 15:07:24.954456 8558 ts_tablet_manager.cc:1939] T 3ea63e471e504999a7c85ef322e6ede0 P 1913adf0f67c403e9667952d11624878: Deleting consensus metadata
I20260927 15:07:24.955886 7973 catalog_manager.cc:4841] TS 1913adf0f67c403e9667952d11624878 (127.0.97.1:46585): tablet 3ea63e471e504999a7c85ef322e6ede0 (table test-table [id=c9a3e9d853a54cd3bbb09dca318fa84b]) successfully deleted
Sep 27 15:07:24 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 15:07:24.972419 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.941652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.0.97.1:46585)
0927 15:07:24.941833 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.941844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.941872 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:07:24.942103 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.942109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.942205 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.942291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.944112 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.945777 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.957489 (+ 11712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.958190 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.958329 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.963837 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.963856 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.963877 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.963933 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.968512 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.969509 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.969518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.969522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.969783 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.970253 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.970259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.970263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.970823 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.971288 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.971624 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.971894 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:07:24.973263 8189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:25.170718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 8058
I20260927 15:07:25.198508 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 8198
Sep 27 15:07:25 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 15:07:25.215385 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.199762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.97.2:46245)
0927 15:07:25.199971 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.199979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.199999 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:07:25.200148 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.200152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.200253 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.200514 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:25.201879 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.202647 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.204044 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.204487 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.204643 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.209297 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:25.209310 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:25.209324 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:25.209368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:25.212339 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.213116 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.213124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.213126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.213281 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.213712 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.213717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.213720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.214125 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:25.214455 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.214779 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.215006 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:07:25.216034 8328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:25.383219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 8198
I20260927 15:07:25.409788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 8337
Sep 27 15:07:25 dist-test-slave-g0w1 krb5kdc[7942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 15:07:25.427018 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.410532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.0.97.3:37971)
0927 15:07:25.411563 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.411568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.411584 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:07:25.411638 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.411641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.411699 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.411831 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:25.413150 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.413860 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.415161 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.415608 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.415754 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.420398 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:25.420415 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:25.420432 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:25.420473 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:25.423570 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.424297 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.424305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.424307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.424447 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.424866 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.424870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.424873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.425267 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:25.425601 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.426275 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.426735 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":823,"threads_started":1}
I20260927 15:07:25.427986 8468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:25.554526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 8337
I20260927 15:07:25.576994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 7955
I20260927 15:07:25.578346 8027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:25.695742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 7955
2026-09-27T15:07:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6173 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:25 dist-test-slave-g0w1 krb5kdc[8615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:25 dist-test-slave-g0w1 krb5kdc[8615](info): set up 2 sockets
Sep 27 15:07:25 dist-test-slave-g0w1 krb5kdc[8615](info): commencing operation
krb5kdc: starting...
W20260927 15:07:26.832007 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:26 dist-test-slave-g0w1 krb5kdc[8615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:07:27.067042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33485
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:27.405290 8628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:27.405679 8628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:27.405853 8628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:27.413358 8628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:27.413529 8628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:27.413648 8628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:27.413750 8628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:27.413841 8628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:27.423970 8628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:27.425369 8628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:27.426760 8628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:27.439484 8636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:27.439492 8633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:27.439857 8634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:27.445171 8628 server_base.cc:1034] running on GCE node
I20260927 15:07:27.446116 8628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:27.447872 8628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:27.449136 8628 hybrid_clock.cc:648] HybridClock initialized: now 1790521647449075 us; error 34 us; skew 500 ppm
I20260927 15:07:27.449487 8628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T15:07:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1818 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:27 dist-test-slave-g0w1 krb5kdc[8642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:27 dist-test-slave-g0w1 krb5kdc[8642](info): set up 2 sockets
Sep 27 15:07:27 dist-test-slave-g0w1 krb5kdc[8642](info): commencing operation
krb5kdc: starting...
W20260927 15:07:28.620306 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:28 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:07:28.856673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35567
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:29.194701 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:29.195019 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:29.195220 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:29.202613 8655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:29.202737 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:29.202785 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:29.202834 8655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:29.202881 8655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:29.213214 8655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:29.214510 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:29.215777 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:29.228541 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:29.228581 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:29.228626 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:29.229436 8655 server_base.cc:1034] running on GCE node
I20260927 15:07:29.230304 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:29.232003 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:29.233239 8655 hybrid_clock.cc:648] HybridClock initialized: now 1790521649233178 us; error 56 us; skew 500 ppm
I20260927 15:07:29.233630 8655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:29 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:29.243992 8655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:07:29.245687 8655 webserver.cc:460] Webserver started at http://127.0.97.62:46665/ using document root <none> and password file <none>
I20260927 15:07:29.246431 8655 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:29.246541 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:29.246803 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:29.250104 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "6604ff0eb5ca440e9ec4684e9433baf8"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "1676f551cbe809a83b93fc668f8c00ae"
server_key_iv: "a6eb4f2b868eca75fcfb3380b91a9e8c"
server_key_version: "encryptionkey@0"
I20260927 15:07:29.250910 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "6604ff0eb5ca440e9ec4684e9433baf8"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "1676f551cbe809a83b93fc668f8c00ae"
server_key_iv: "a6eb4f2b868eca75fcfb3380b91a9e8c"
server_key_version: "encryptionkey@0"
I20260927 15:07:29.256486 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 15:07:29.260376 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:29.261902 8655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:29.262110 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "6604ff0eb5ca440e9ec4684e9433baf8"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "1676f551cbe809a83b93fc668f8c00ae"
server_key_iv: "a6eb4f2b868eca75fcfb3380b91a9e8c"
server_key_version: "encryptionkey@0"
I20260927 15:07:29.262331 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:29.287294 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:29.291908 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:29.292227 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:29.311946 8655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35567
I20260927 15:07:29.311998 8721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35567 every 8 connection(s)
I20260927 15:07:29.313462 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:29.317627 8722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:29.319902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 8655
I20260927 15:07:29.320123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:29.320766 388 external_mini_cluster.cc:1428] Setting key 3c5cdf7be1c2238211b9d64ca5a62a84
I20260927 15:07:29.330271 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Bootstrap starting.
I20260927 15:07:29.334474 8722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:29.336025 8722 log.cc:824] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:07:29 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:29.340323 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: No bootstrap required, opened a new log
I20260927 15:07:29.348400 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:29.323158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.0.97.62:35567)
0927 15:07:29.323977 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:29.323990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:29.324035 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:07:29.325737 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:29.325763 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:29.326212 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:29.326613 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:29.328622 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:29.330240 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:29.332245 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:29.333044 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:29.333211 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:29.338616 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:29.338627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:29.338650 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:29.338701 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:29.342913 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:29.343912 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:29.343927 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:29.343938 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:29.344173 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:29.344618 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:29.344624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:29.344627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:29.345424 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:29.345784 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:29.346218 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 15:07:29.346786 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":372,"threads_started":1}
I20260927 15:07:29.351375 8722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:29.351868 8722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:29.351981 8722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6604ff0eb5ca440e9ec4684e9433baf8, State: Initialized, Role: FOLLOWER
I20260927 15:07:29.352543 8722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:29.352782 8722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:29.352911 8722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:29.353096 8722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:29.355358 8722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:29.355906 8722 leader_election.cc:302] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6604ff0eb5ca440e9ec4684e9433baf8; no voters:
I20260927 15:07:29.356592 8722 leader_election.cc:288] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:29.356794 8727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:29.358263 8727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 LEADER]: Becoming Leader. State: Replica: 6604ff0eb5ca440e9ec4684e9433baf8, State: Running, Role: LEADER
I20260927 15:07:29.358847 8722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:29.358819 8727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:29.362807 8728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } } }
I20260927 15:07:29.362769 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6604ff0eb5ca440e9ec4684e9433baf8. Latest consensus state: current_term: 1 leader_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } } }
I20260927 15:07:29.363487 8728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:29.363530 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:29.364583 8736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:29.373529 8736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:29.386968 8736 catalog_manager.cc:1347] Generated new cluster ID: 5d3e0d91a75b4260a934d2d87effa38a
I20260927 15:07:29.387115 8736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:29.409467 8736 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:29.411015 8736 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:29.420650 8736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Generated new TSK 0
I20260927 15:07:29.421672 8736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:07:29.479839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:29.835901 8748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:29.836265 8748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:29.836433 8748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:29.844421 8748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:29.844592 8748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:29.844763 8748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:29.855584 8748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:29.856984 8748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:29.858417 8748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:29.874336 8753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:29.874286 8754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:29.874414 8756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:29.875284 8748 server_base.cc:1034] running on GCE node
I20260927 15:07:29.876019 8748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:29.877179 8748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:29.878441 8748 hybrid_clock.cc:648] HybridClock initialized: now 1790521649878394 us; error 53 us; skew 500 ppm
I20260927 15:07:29.878810 8748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:29 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:29.882540 8748 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:07:29.884298 8748 webserver.cc:460] Webserver started at http://127.0.97.1:38939/ using document root <none> and password file <none>
I20260927 15:07:29.885165 8748 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:29.885320 8748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:29.885648 8748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:29.888970 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "9e90748c7af54fcb9f1f0742d76bcce7"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "7f9a65999a6e26f804aab4eff57295dd"
server_key_iv: "bd7a299091db6f9b5b98ab038d48454b"
server_key_version: "encryptionkey@0"
I20260927 15:07:29.889863 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9e90748c7af54fcb9f1f0742d76bcce7"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "7f9a65999a6e26f804aab4eff57295dd"
server_key_iv: "bd7a299091db6f9b5b98ab038d48454b"
server_key_version: "encryptionkey@0"
I20260927 15:07:29.895710 8748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:07:29.899672 8763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:29.901255 8748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:07:29.901475 8748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "9e90748c7af54fcb9f1f0742d76bcce7"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "7f9a65999a6e26f804aab4eff57295dd"
server_key_iv: "bd7a299091db6f9b5b98ab038d48454b"
server_key_version: "encryptionkey@0"
I20260927 15:07:29.901706 8748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:29.939478 8748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:29.944483 8748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:29.944818 8748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:29.945932 8748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:29.947433 8748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:29.947539 8748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:29.947707 8748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:29.947780 8748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:29.992702 8748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42529
I20260927 15:07:29.992753 8875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42529 every 8 connection(s)
I20260927 15:07:29.994059 8748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:29.997300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 8748
I20260927 15:07:29.997534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:29.998174 388 external_mini_cluster.cc:1428] Setting key 55b04fb3b0440cd22e809ec5df58bff7
Sep 27 15:07:30 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:30.018709 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:29.996892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33053 (local address 127.0.97.62:35567)
0927 15:07:29.997442 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:29.997451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:29.998828 (+ 1377us) server_negotiation.cc:407] Connection header received
0927 15:07:30.000791 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:30.000796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:30.000889 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:30.001017 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:30.002757 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.003946 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.005819 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.006724 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.006948 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:30.012280 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:30.012287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:30.012290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:30.012335 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:30.015331 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:30.016215 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:30.016220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:30.016223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:30.016352 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:30.016818 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:30.016826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:30.016831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:30.017108 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:30.017268 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:30.018081 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 15:07:30.018344 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":265,"threads_started":1}
I20260927 15:07:30.019873 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:29.997700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:33053)
0927 15:07:29.998615 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:29.998677 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:30.000450 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:30.001294 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:30.001304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:30.001897 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:30.002539 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:30.002559 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.004138 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.004143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:30.005646 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:30.005662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.007419 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.007435 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:30.007833 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:30.008742 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:30.008781 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:30.011974 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:30.015510 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:30.015523 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:30.015543 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:30.016052 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:30.016476 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:30.016481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:30.016485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:30.016695 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:30.017252 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:30.017262 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:30.017656 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 15:07:30.017970 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 15:07:30.018568 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":412,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:07:30.022185 8876 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35567
I20260927 15:07:30.022601 8876 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:30.023540 8876 heartbeater.cc:511] Master 127.0.97.62:35567 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:07:30.026105 8687 ts_manager.cc:194] Registered new tserver with Master: 9e90748c7af54fcb9f1f0742d76bcce7 (127.0.97.1:42529)
I20260927 15:07:30.027782 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:07:30.056187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:30.412196 8882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:30.412510 8882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:30.412693 8882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:30.420331 8882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:30.420451 8882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:30.420640 8882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:30.430876 8882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:30.432315 8882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:30.433683 8882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:30.448163 8890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:30.448093 8887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:30.448211 8888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:30.453047 8882 server_base.cc:1034] running on GCE node
I20260927 15:07:30.453738 8882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:30.454830 8882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:30.456059 8882 hybrid_clock.cc:648] HybridClock initialized: now 1790521650455998 us; error 52 us; skew 500 ppm
I20260927 15:07:30.456432 8882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:30 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:30.459940 8882 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:07:30.461790 8882 webserver.cc:460] Webserver started at http://127.0.97.2:42051/ using document root <none> and password file <none>
I20260927 15:07:30.462580 8882 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:30.462713 8882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:30.463145 8882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:30.466259 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "abbe1eae9f9848fab412788ad7f53389"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "6b9e3656d9efe64d43679bfed6fc338c"
server_key_iv: "9f07c4bffc7c52ee5b750f84a643bf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.467188 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "abbe1eae9f9848fab412788ad7f53389"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "6b9e3656d9efe64d43679bfed6fc338c"
server_key_iv: "9f07c4bffc7c52ee5b750f84a643bf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.473002 8882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:30.476984 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:30.478575 8882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:30.478775 8882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "abbe1eae9f9848fab412788ad7f53389"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "6b9e3656d9efe64d43679bfed6fc338c"
server_key_iv: "9f07c4bffc7c52ee5b750f84a643bf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.479027 8882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:30.494453 8882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:30.499020 8882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:30.499426 8882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:30.500478 8882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:30.501946 8882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:30.502063 8882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:30.502219 8882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:30.502311 8882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:30.537842 8882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46777
I20260927 15:07:30.537889 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46777 every 8 connection(s)
I20260927 15:07:30.539345 8882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:30.541772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 8882
I20260927 15:07:30.541992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:30.542593 388 external_mini_cluster.cc:1428] Setting key 41b41c7cf3c5cc67694db1d4fcd619a6
Sep 27 15:07:30 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:30.563172 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:30.542104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37233 (local address 127.0.97.62:35567)
0927 15:07:30.542563 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:30.542570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:30.543875 (+ 1305us) server_negotiation.cc:407] Connection header received
0927 15:07:30.545980 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:30.545988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:30.546085 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:30.546215 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:30.548181 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.549403 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.551265 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.551711 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.551879 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:30.555883 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:30.555892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:30.555900 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:30.555949 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:30.559640 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:30.560643 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:30.560651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:30.560658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:30.560824 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:30.561282 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:30.561296 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:30.561299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:30.561589 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:30.561755 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:30.562516 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 15:07:30.562744 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260927 15:07:30.564196 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:30.542843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:37233)
0927 15:07:30.543707 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:30.543764 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:30.545675 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:30.546486 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:30.546497 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:30.547345 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:30.547976 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:30.547997 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.549653 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.549658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:30.551018 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:30.551032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.551982 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.551990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:30.552247 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:30.553027 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:30.553052 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:30.555583 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:30.559896 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:30.559905 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:30.559922 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:30.560489 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:30.560936 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:30.560940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:30.560942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:30.561141 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:30.561732 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:30.561741 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:30.562123 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 15:07:30.562402 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 15:07:30.563018 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":380,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:07:30.565781 9010 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35567
I20260927 15:07:30.566107 9010 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:30.566934 9010 heartbeater.cc:511] Master 127.0.97.62:35567 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:07:30.568717 8687 ts_manager.cc:194] Registered new tserver with Master: abbe1eae9f9848fab412788ad7f53389 (127.0.97.2:46777)
I20260927 15:07:30.569607 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:07:30.595290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:30.940348 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:30.940651 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:30.940827 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:30.948485 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:30.948613 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:30.948845 9016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:30.959308 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:30.960902 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:30.962275 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:30.976308 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:30.976456 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:30.976620 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:30.981019 9016 server_base.cc:1034] running on GCE node
I20260927 15:07:30.981769 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:30.982983 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:30.984253 9016 hybrid_clock.cc:648] HybridClock initialized: now 1790521650984178 us; error 76 us; skew 500 ppm
I20260927 15:07:30.984640 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:30 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:30.988277 9016 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:07:30.990061 9016 webserver.cc:460] Webserver started at http://127.0.97.3:40301/ using document root <none> and password file <none>
I20260927 15:07:30.990871 9016 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:30.991030 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:30.991418 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:30.994634 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "94d21f19bcaf4ddea094505473c53d57"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "20d226222f070770549098b01983f19e"
server_key_iv: "dc867cb4b0a88d58a688620051cedb3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.995549 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "94d21f19bcaf4ddea094505473c53d57"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "20d226222f070770549098b01983f19e"
server_key_iv: "dc867cb4b0a88d58a688620051cedb3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.001380 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:07:31.005475 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.007336 9016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:31.007555 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "94d21f19bcaf4ddea094505473c53d57"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "20d226222f070770549098b01983f19e"
server_key_iv: "dc867cb4b0a88d58a688620051cedb3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.007817 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:31.031886 8876 heartbeater.cc:503] Master 127.0.97.62:35567 was elected leader, sending a full tablet report...
I20260927 15:07:31.037109 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:31.042107 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:31.042428 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:31.043599 9016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:31.045104 9016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:31.045219 9016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.045377 9016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:31.045459 9016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.081701 9016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43489
I20260927 15:07:31.081759 9143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43489 every 8 connection(s)
I20260927 15:07:31.083190 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:31.088845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9016
I20260927 15:07:31.089069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:31.089742 388 external_mini_cluster.cc:1428] Setting key 0af80c08052d2d5a7ebab29a33a9dbb4
Sep 27 15:07:31 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:31.106624 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.086294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52275 (local address 127.0.97.62:35567)
0927 15:07:31.086753 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.086758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.087945 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 15:07:31.090720 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.090726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.090811 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.090932 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.093027 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.093872 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.095760 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.096214 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.096363 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.100264 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.100269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.100272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.100309 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.103137 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.104028 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.104035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.104040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.104183 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.104661 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.104667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.104671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.104984 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.105158 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.105982 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.106223 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":209,"threads_started":1}
I20260927 15:07:31.107801 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.086895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:52275)
0927 15:07:31.087752 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:31.087801 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:31.090411 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:31.091294 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:31.091305 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:31.091911 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:31.092823 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.092854 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.094047 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.094053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.095592 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.095605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.096469 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.096477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.096742 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.097525 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:31.097551 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:31.099996 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:31.103315 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.103325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.103347 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.103881 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.104310 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.104317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.104321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.104536 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.105137 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:31.105146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:31.105528 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 15:07:31.105872 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 15:07:31.106474 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":369,"threads_started":1}
I20260927 15:07:31.109264 9144 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35567
I20260927 15:07:31.109544 9144 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:31.110303 9144 heartbeater.cc:511] Master 127.0.97.62:35567 requested a full tablet report, sending...
I20260927 15:07:31.111831 8687 ts_manager.cc:194] Registered new tserver with Master: 94d21f19bcaf4ddea094505473c53d57 (127.0.97.3:43489)
I20260927 15:07:31.112661 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52275
I20260927 15:07:31.118014 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 15:07:31.261938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 8748
Sep 27 15:07:31 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:07:31.280224 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.263290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.0.97.1:42529)
0927 15:07:31.264065 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.264073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.264102 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:07:31.264164 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.264171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.264456 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.264599 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.266036 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.267219 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.268518 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.269094 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.269216 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.273921 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.273937 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.273953 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.273997 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.277144 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.277922 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.277931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.277934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.278093 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.278421 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.278424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.278426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.278975 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.279453 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.279680 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.279909 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":422,"threads_started":1}
I20260927 15:07:31.281571 8871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:31.364871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 8748
I20260927 15:07:31.381474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 8882
Sep 27 15:07:31 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:07:31.398589 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.382461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.0.97.2:46777)
0927 15:07:31.383034 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.383041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.383126 (+ 85us) server_negotiation.cc:407] Connection header received
0927 15:07:31.383207 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.383219 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.383474 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.383611 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.385018 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.386068 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.387514 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.388092 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.388198 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.392603 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.392616 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.392633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.392669 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.395702 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.396516 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.396523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.396525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.396650 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.397052 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.397055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.397058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.397406 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.397649 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.397995 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.398284 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":236,"threads_started":1}
I20260927 15:07:31.400049 9005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:31.484746 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 8882
I20260927 15:07:31.501699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9016
Sep 27 15:07:31 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:07:31.519766 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.502766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.3:43489)
0927 15:07:31.503572 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.503579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.503609 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:07:31.503670 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.503676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.503893 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.504026 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.505499 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.506580 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.507964 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.508538 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.508651 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.513532 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.513544 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.513559 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.513601 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.516574 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.517294 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.517302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.517305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.517463 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.517929 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.517934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.517936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.518326 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.518571 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.518994 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.519320 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":387,"threads_started":1}
I20260927 15:07:31.521210 9139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:31.604739 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9016
I20260927 15:07:31.621510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 8655
I20260927 15:07:31.622819 8717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:31.731376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 8655
I20260927 15:07:31.752678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35567
--webserver_interface=127.0.97.62
--webserver_port=46665
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:32.106659 9162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:32.106980 9162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:32.107131 9162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:32.114722 9162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:32.114846 9162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:32.114893 9162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:32.114946 9162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:32.114993 9162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:32.125345 9162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=46665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:32.126652 9162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:32.127964 9162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:32.140393 9170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:32.140486 9167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:32.140410 9168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:32.141176 9162 server_base.cc:1034] running on GCE node
I20260927 15:07:32.142011 9162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:32.143658 9162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:32.144881 9162 hybrid_clock.cc:648] HybridClock initialized: now 1790521652144828 us; error 43 us; skew 500 ppm
I20260927 15:07:32.145272 9162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:32.148595 9162 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:07:32.150555 9162 webserver.cc:460] Webserver started at http://127.0.97.62:46665/ using document root <none> and password file <none>
I20260927 15:07:32.151479 9162 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:32.151603 9162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:32.157404 9162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:32.160645 9177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:32.162334 9162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:32.162544 9162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "6604ff0eb5ca440e9ec4684e9433baf8"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "1676f551cbe809a83b93fc668f8c00ae"
server_key_iv: "a6eb4f2b868eca75fcfb3380b91a9e8c"
server_key_version: "encryptionkey@0"
I20260927 15:07:32.163656 9162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:32.187702 9162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:32.192411 9162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:32.192713 9162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:32.212211 9162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35567
I20260927 15:07:32.212229 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35567 every 8 connection(s)
I20260927 15:07:32.213927 9162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:32.216230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9162
I20260927 15:07:32.217008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42529
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35567
--webserver_port=38939
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:32.221446 9229 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:07:32.226661 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Bootstrap starting.
I20260927 15:07:32.256472 9229 log.cc:824] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:32.268214 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:07:32.268884 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Bootstrap complete.
I20260927 15:07:32.282846 9229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:32.284153 9229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6604ff0eb5ca440e9ec4684e9433baf8, State: Initialized, Role: FOLLOWER
I20260927 15:07:32.285063 9229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:32.285429 9229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:32.285581 9229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:32.285811 9229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:07:32.292511 9229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:32.293234 9229 leader_election.cc:302] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6604ff0eb5ca440e9ec4684e9433baf8; no voters:
I20260927 15:07:32.293936 9229 leader_election.cc:288] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:07:32.294114 9233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:07:32.295379 9233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [term 2 LEADER]: Becoming Leader. State: Replica: 6604ff0eb5ca440e9ec4684e9433baf8, State: Running, Role: LEADER
I20260927 15:07:32.296052 9233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260927 15:07:32.296201 9229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:32.298465 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6604ff0eb5ca440e9ec4684e9433baf8. Latest consensus state: current_term: 2 leader_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } } }
I20260927 15:07:32.298854 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:32.299441 9234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6604ff0eb5ca440e9ec4684e9433baf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } } }
I20260927 15:07:32.299811 9234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:32.310248 9239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:32.316527 9239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:32.318106 9239 catalog_manager.cc:1259] Loaded cluster ID: 5d3e0d91a75b4260a934d2d87effa38a
I20260927 15:07:32.318324 9239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:32.322623 9239 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:32.324357 9239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6604ff0eb5ca440e9ec4684e9433baf8: Loaded TSK: 0
I20260927 15:07:32.326107 9239 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:07:32.608363 9231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:32.608662 9231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:32.608814 9231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:32.616667 9231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:32.616806 9231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:32.616981 9231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:32.627445 9231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38939
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:32.628852 9231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:32.630283 9231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:32.645232 9255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:32.645334 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:32.645502 9258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:32.645926 9231 server_base.cc:1034] running on GCE node
I20260927 15:07:32.646792 9231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:32.648154 9231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:32.649439 9231 hybrid_clock.cc:648] HybridClock initialized: now 1790521652649374 us; error 63 us; skew 500 ppm
I20260927 15:07:32.649858 9231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521652, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:32.654687 9231 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 15:07:32.657001 9231 webserver.cc:460] Webserver started at http://127.0.97.1:38939/ using document root <none> and password file <none>
I20260927 15:07:32.658003 9231 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:32.658172 9231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:32.666532 9231 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:07:32.670553 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:32.672559 9231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:07:32.672771 9231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "9e90748c7af54fcb9f1f0742d76bcce7"
format_stamp: "Formatted at 2026-09-27 15:07:29 on dist-test-slave-g0w1"
server_key: "7f9a65999a6e26f804aab4eff57295dd"
server_key_iv: "bd7a299091db6f9b5b98ab038d48454b"
server_key_version: "encryptionkey@0"
I20260927 15:07:32.674407 9231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:32.690815 9231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:32.695219 9231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:32.695453 9231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:32.696590 9231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:32.698064 9231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:32.698164 9231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:32.698238 9231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:32.698295 9231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:32.733479 9231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42529
I20260927 15:07:32.733578 9377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42529 every 8 connection(s)
I20260927 15:07:32.735033 9231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:32.745287 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9231
I20260927 15:07:32.746025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46777
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35567
--webserver_port=42051
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:32.756871 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:32.738500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:49237)
0927 15:07:32.739418 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:32.739460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:32.741151 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:32.742641 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:32.742650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:32.743314 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:32.743888 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:32.743906 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.746350 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.746358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:32.748689 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:32.748711 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.749914 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.749923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:32.750257 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.751126 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:32.751163 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:32.755871 (+ 4708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":381,"threads_started":1}
W20260927 15:07:32.757582 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:32.738346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49237 (local address 127.0.97.62:35567)
0927 15:07:32.739192 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:32.739211 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:32.739602 (+ 391us) server_negotiation.cc:407] Connection header received
0927 15:07:32.741753 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:32.741769 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:32.742140 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:32.742489 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:32.744117 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.746108 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.748894 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.749650 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.749816 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.756363 (+ 6547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49237: BlockingRecv error: recv got EOF from 127.0.97.1:49237 (error 108)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":285,"threads_started":1}
W20260927 15:07:32.757694 9378 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35567 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35567: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:32.766521 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:32.758310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:38409)
0927 15:07:32.758480 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:32.758498 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:32.758635 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:32.758961 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:32.758966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:32.759337 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:32.759492 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:32.759500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.760536 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.760542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:32.761446 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:32.761458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.762168 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.762175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:32.762360 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.763001 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:32.763019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:32.766295 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:07:32.766902 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:32.758468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38409 (local address 127.0.97.62:35567)
0927 15:07:32.758640 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:32.758645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:32.758665 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:07:32.758754 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:32.758758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:32.758825 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:32.758935 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:32.759634 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.760386 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.761603 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.762009 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.762149 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.766613 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38409: BlockingRecv error: recv got EOF from 127.0.97.1:38409 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:32 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:32.775487 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:32.767308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:59467)
0927 15:07:32.767489 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:32.767514 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:32.767640 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:32.768050 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:32.768054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:32.768440 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:32.768604 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:32.768612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.769666 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.769672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:32.770595 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:32.770607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.771407 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.771413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:32.771588 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.772226 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:32.772250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:32.775150 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:07:32.775661 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:32.767518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59467 (local address 127.0.97.62:35567)
0927 15:07:32.767702 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:32.767708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:32.767728 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:07:32.767801 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:32.767806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:32.767902 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:32.768015 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:32.768755 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.769517 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.770764 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.771245 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.771381 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.775401 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59467: BlockingRecv error: recv got EOF from 127.0.97.1:59467 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 15:07:32.775866 9378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:07:33.098557 9382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:33.098915 9382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:33.099121 9382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:33.106694 9382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:33.106879 9382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:33.107096 9382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:33.117445 9382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42051
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:33.118888 9382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:33.120373 9382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:33.134305 9387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:33.134420 9390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:33.134425 9388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:33.135327 9382 server_base.cc:1034] running on GCE node
I20260927 15:07:33.135999 9382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:33.137056 9382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:33.138329 9382 hybrid_clock.cc:648] HybridClock initialized: now 1790521653138286 us; error 40 us; skew 500 ppm
I20260927 15:07:33.138723 9382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:33.142321 9382 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 15:07:33.144042 9382 webserver.cc:460] Webserver started at http://127.0.97.2:42051/ using document root <none> and password file <none>
I20260927 15:07:33.144836 9382 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:33.145008 9382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:33.150992 9382 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:33.154152 9397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:33.156068 9382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:33.156299 9382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "abbe1eae9f9848fab412788ad7f53389"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "6b9e3656d9efe64d43679bfed6fc338c"
server_key_iv: "9f07c4bffc7c52ee5b750f84a643bf29"
server_key_version: "encryptionkey@0"
I20260927 15:07:33.157335 9382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:33.179301 9382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:33.184007 9382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:33.184353 9382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:33.185431 9382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:33.186916 9382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:33.187059 9382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:33.187278 9382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:33.187368 9382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:33.223176 9382 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46777
I20260927 15:07:33.223373 9509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46777 every 8 connection(s)
I20260927 15:07:33.225042 9382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:33.232832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9382
I20260927 15:07:33.233553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43489
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35567
--webserver_port=40301
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:33.244766 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.228182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44117 (local address 127.0.97.62:35567)
0927 15:07:33.228393 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.228398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.229831 (+ 1433us) server_negotiation.cc:407] Connection header received
0927 15:07:33.231874 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.231880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.231978 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.232103 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.234468 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.235279 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.237918 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.238383 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.238491 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.244509 (+ 6018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44117: BlockingRecv error: recv got EOF from 127.0.97.2:44117 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:07:33.245566 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.228803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:44117)
0927 15:07:33.229631 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.229672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.231561 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.232417 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.232431 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.233245 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.234169 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.234201 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.235467 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.235475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.237723 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.237743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.238661 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.238671 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.239031 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.239860 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.239903 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.244317 (+ 4414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":339,"threads_started":1}
W20260927 15:07:33.246284 9510 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35567 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35567: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:33.256175 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.246929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:38859)
0927 15:07:33.247145 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.247162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.247264 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.247622 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.247625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.247946 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.248084 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.248092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.249716 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.249721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.250778 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.250797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.252025 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.252030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.252169 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.252819 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.252838 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.255957 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:07:33.256618 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.247035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38859 (local address 127.0.97.62:35567)
0927 15:07:33.247271 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.247279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.247339 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:07:33.247404 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.247412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.247497 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.247597 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.248235 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.249537 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.250973 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.251842 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.252008 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.256324 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38859: BlockingRecv error: recv got EOF from 127.0.97.2:38859 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:33.266110 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.257083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:42817)
0927 15:07:33.257243 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.257263 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.257373 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.257756 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.257760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.258033 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.258186 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.258194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.259747 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.259751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.260766 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.260784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.261697 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.261702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.261929 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.262612 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.262630 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.265886 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:07:33.266394 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.257184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42817 (local address 127.0.97.62:35567)
0927 15:07:33.257387 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.257395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.257418 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:07:33.257492 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.257500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.257590 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.257710 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.258354 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.259581 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.260927 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.261552 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.261675 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.266166 (+ 4491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42817: BlockingRecv error: recv got EOF from 127.0.97.2:42817 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 15:07:33.266520 9510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:07:33.585383 9513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:33.585703 9513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:33.585799 9513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:33.593384 9513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:33.593528 9513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:33.593644 9513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:33.603904 9513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40301
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:33.605221 9513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:33.606498 9513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:33.620523 9518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:33.620851 9519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:33.621116 9521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:33.621605 9513 server_base.cc:1034] running on GCE node
I20260927 15:07:33.622272 9513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:33.623401 9513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:33.624701 9513 hybrid_clock.cc:648] HybridClock initialized: now 1790521653624649 us; error 50 us; skew 500 ppm
I20260927 15:07:33.625062 9513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521653, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:33.628710 9513 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 15:07:33.630561 9513 webserver.cc:460] Webserver started at http://127.0.97.3:40301/ using document root <none> and password file <none>
I20260927 15:07:33.631464 9513 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:33.631649 9513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:33.637651 9513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 15:07:33.640869 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:33.642463 9513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:33.642680 9513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "94d21f19bcaf4ddea094505473c53d57"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-g0w1"
server_key: "20d226222f070770549098b01983f19e"
server_key_iv: "dc867cb4b0a88d58a688620051cedb3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:33.643814 9513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:33.665261 9513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:33.670228 9513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:33.670528 9513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:33.671626 9513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:33.673118 9513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:33.673264 9513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:33.673390 9513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:33.673475 9513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:33.708189 9513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43489
I20260927 15:07:33.708293 9640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43489 every 8 connection(s)
I20260927 15:07:33.709740 9513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:33.710465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9513
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:33.729967 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.713222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38829 (local address 127.0.97.62:35567)
0927 15:07:33.713823 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.713829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.715091 (+ 1262us) server_negotiation.cc:407] Connection header received
0927 15:07:33.717467 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.717471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.717534 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.717619 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.719776 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.720549 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.722538 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.723033 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.723214 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.729605 (+ 6391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38829: BlockingRecv error: recv got EOF from 127.0.97.3:38829 (error 108)
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
I20260927 15:07:33.730013 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.711622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.0.97.62:35567)
0927 15:07:33.711820 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.711826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.712922 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 15:07:33.713148 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.713155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.713258 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.713372 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.715392 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.716102 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.717988 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.718386 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.718480 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.721601 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:33.721609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:33.721626 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:33.721667 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:33.725703 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:33.726660 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:33.726670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:33.726681 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:33.726939 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:33.727650 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:33.727656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:33.727659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:33.728372 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:33.728754 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:33.729253 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 15:07:33.729679 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:07:33.730648 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.713947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:38829)
0927 15:07:33.714850 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.714906 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.717134 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.717951 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.717966 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.718649 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.719558 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.719587 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.720744 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.720750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.722378 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.722391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.723343 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.723355 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.723693 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.724535 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.724567 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.729370 (+ 4803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":341,"threads_started":1}
W20260927 15:07:33.731506 9641 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35567 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35567: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:33.741246 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.732134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:57313)
0927 15:07:33.732320 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.732346 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.732516 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.732924 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.732930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.733264 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.733420 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.733428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.734850 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.734855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.736205 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.736220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.737144 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.737150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.737346 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.738005 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.738024 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.741020 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:07:33.741475 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.732246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57313 (local address 127.0.97.62:35567)
0927 15:07:33.732421 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.732428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.732449 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:07:33.732661 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.732666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.732746 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.732849 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.733567 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.734680 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.736359 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.736981 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.737118 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.741221 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57313: BlockingRecv error: recv got EOF from 127.0.97.3:57313 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:33.750095 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.741968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:41611)
0927 15:07:33.742171 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.742192 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.742297 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.742566 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.742569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.742848 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.742995 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.743003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.744094 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.744098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.745026 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.745037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.745787 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.745791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.745919 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.746482 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.746500 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.749880 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:07:33.750309 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.742058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41611 (local address 127.0.97.62:35567)
0927 15:07:33.742289 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.742294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.742311 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:07:33.742390 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.742394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.742450 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.742529 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.743194 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.743972 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.745207 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.745662 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.745760 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.750060 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41611: BlockingRecv error: recv got EOF from 127.0.97.3:41611 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
W20260927 15:07:33.750415 9641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:33 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:33.785815 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:33.776523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:50687)
0927 15:07:33.777095 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:33.777111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:33.777229 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:33.777621 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:33.777625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:33.778109 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:33.778317 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.778326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.779604 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.779608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.780714 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:33.780726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.781619 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.781624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:33.781763 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.782393 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:33.782411 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:33.785477 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":319,"threads_started":1}
W20260927 15:07:33.786027 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:33.776617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50687 (local address 127.0.97.62:35567)
0927 15:07:33.776839 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:33.776845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:33.777215 (+ 370us) server_negotiation.cc:407] Connection header received
0927 15:07:33.777393 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:33.777398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:33.777470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:33.777575 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:33.778481 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.779447 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.780907 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:33.781435 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:33.781541 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:33.785751 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50687: BlockingRecv error: recv got EOF from 127.0.97.1:50687 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:07:34 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:34 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:34.282560 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.267213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:47613)
0927 15:07:34.267701 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:34.267718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:34.267851 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:34.268195 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:34.268202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:34.268727 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:34.268936 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.268947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.270078 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.270082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.271103 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.271117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.271942 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.271947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.272080 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.279003 (+ 6923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:34.279034 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:34.282311 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
W20260927 15:07:34.282825 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:34.267242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47613 (local address 127.0.97.62:35567)
0927 15:07:34.267452 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:34.267461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:34.267840 (+ 379us) server_negotiation.cc:407] Connection header received
0927 15:07:34.267947 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:34.267951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:34.268015 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:34.268108 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:34.269114 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.269905 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.271295 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.271726 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.271837 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.282578 (+ 10741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47613: BlockingRecv error: recv got EOF from 127.0.97.2:47613 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 15:07:34 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:34 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:34.760607 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.751083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:56635)
0927 15:07:34.751641 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:34.751657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:34.751821 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:34.752182 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:34.752187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:34.752669 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:34.752890 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.752900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.754033 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.754040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.755050 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.755064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.755999 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.756004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.756223 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.757140 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:34.757165 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:34.760367 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":279,"threads_started":1}
W20260927 15:07:34.760823 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:34.751243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56635 (local address 127.0.97.62:35567)
0927 15:07:34.751513 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:34.751519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:34.751749 (+ 230us) server_negotiation.cc:407] Connection header received
0927 15:07:34.751968 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:34.751975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:34.752044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:34.752150 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:34.753057 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.753881 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.755297 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.755773 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.755903 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.760590 (+ 4687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56635: BlockingRecv error: recv got EOF from 127.0.97.3:56635 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 27 15:07:34 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:34 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:34.796370 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.786998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:40089)
0927 15:07:34.787730 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:34.787747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:34.787888 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:34.788256 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:34.788259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:34.788637 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:34.788784 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.788792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.789808 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.789812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.790723 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.790735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.791617 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.791622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.791744 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.792450 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:34.792469 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:34.795998 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":497,"threads_started":1}
W20260927 15:07:34.796458 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:34.787132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40089 (local address 127.0.97.62:35567)
0927 15:07:34.787354 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:34.787361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:34.787826 (+ 465us) server_negotiation.cc:407] Connection header received
0927 15:07:34.788041 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:34.788045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:34.788119 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:34.788212 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:34.788945 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.789691 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.790913 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.791429 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.791540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.796207 (+ 4667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40089: BlockingRecv error: recv got EOF from 127.0.97.1:40089 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 15:07:35 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:35 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:35.292778 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:35.283676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:37561)
0927 15:07:35.284273 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:35.284291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:35.284474 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:35.284868 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:35.284873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:35.285287 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:35.285450 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.285460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.286539 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.286546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.287560 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.287572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.288456 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.288462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.288651 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.289492 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:35.289516 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:35.292543 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":330,"threads_started":1}
W20260927 15:07:35.292996 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:35.283867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37561 (local address 127.0.97.62:35567)
0927 15:07:35.284134 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:35.284140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:35.284384 (+ 244us) server_negotiation.cc:407] Connection header received
0927 15:07:35.284620 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:35.284630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:35.284705 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:35.284841 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:35.285606 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.286398 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.287708 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.288222 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.288373 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.292777 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37561: BlockingRecv error: recv got EOF from 127.0.97.2:37561 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 15:07:35 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:35 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:35.771153 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:35.761647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:40947)
0927 15:07:35.762197 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:35.762212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:35.762416 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:35.762763 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:35.762768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:35.763267 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:35.763421 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.763430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.764465 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.764470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.765379 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.765390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.766232 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.766237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.766445 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.767297 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:35.767325 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:35.770742 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":289,"threads_started":1}
W20260927 15:07:35.771310 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:35.761848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40947 (local address 127.0.97.62:35567)
0927 15:07:35.762096 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:35.762103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:35.762302 (+ 199us) server_negotiation.cc:407] Connection header received
0927 15:07:35.762569 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:35.762573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:35.762634 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:35.762737 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:35.763576 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.764306 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.765532 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.766072 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.766215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.770999 (+ 4784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40947: BlockingRecv error: recv got EOF from 127.0.97.3:40947 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 15:07:35 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:35 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:35.806224 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:35.797645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:41495)
0927 15:07:35.798197 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:35.798225 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:35.798349 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:35.798664 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:35.798668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:35.799062 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:35.799274 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.799283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.800412 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.800417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.801329 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.801342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.802075 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.802080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.802209 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.802888 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:35.802905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:35.805967 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":308,"threads_started":1}
W20260927 15:07:35.806368 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:35.797690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41495 (local address 127.0.97.62:35567)
0927 15:07:35.797886 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:35.797891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:35.798318 (+ 427us) server_negotiation.cc:407] Connection header received
0927 15:07:35.798457 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:35.798462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:35.798524 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:35.798616 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:35.799437 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.800250 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.801488 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.801942 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.802043 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.806152 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41495: BlockingRecv error: recv got EOF from 127.0.97.1:41495 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 15:07:36 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:36 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:36.302500 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.293807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:51685)
0927 15:07:36.294441 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:36.294469 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:36.294626 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:36.295024 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:36.295028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:36.295438 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:36.295612 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.295621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.296742 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.296748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.297681 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.297695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.298442 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.298448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.298640 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.299383 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:36.299401 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:36.302234 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":298,"threads_started":1}
W20260927 15:07:36.302713 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:36.293898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51685 (local address 127.0.97.62:35567)
0927 15:07:36.294104 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:36.294110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:36.294563 (+ 453us) server_negotiation.cc:407] Connection header received
0927 15:07:36.294772 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:36.294778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:36.294862 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:36.295000 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:36.295766 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.296579 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.297846 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.298284 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.298425 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.302454 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51685: BlockingRecv error: recv got EOF from 127.0.97.2:51685 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 15:07:36 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:36 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:36.780736 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.772267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:36599)
0927 15:07:36.772885 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:36.772902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:36.773042 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:36.773424 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:36.773429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:36.773798 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:36.773964 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.773972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.775010 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.775015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.775979 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.775992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.776675 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.776679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.776812 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.777490 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:36.777509 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:36.780478 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":355,"threads_started":1}
W20260927 15:07:36.780921 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:36.772399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36599 (local address 127.0.97.62:35567)
0927 15:07:36.772627 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:36.772635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:36.772995 (+ 360us) server_negotiation.cc:407] Connection header received
0927 15:07:36.773167 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:36.773171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:36.773261 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:36.773359 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:36.774148 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.774859 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.776123 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.776532 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.776632 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.780706 (+ 4074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36599: BlockingRecv error: recv got EOF from 127.0.97.3:36599 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 15:07:36 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:36 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:36.816039 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.807376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:40775)
0927 15:07:36.807850 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:36.807868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:36.807999 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:36.808436 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:36.808441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:36.808873 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:36.809035 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.809044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.810118 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.810124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.811200 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.811213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.811955 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.811960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.812114 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.812815 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:36.812833 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:36.815787 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":276,"threads_started":1}
W20260927 15:07:36.816459 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:36.807479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40775 (local address 127.0.97.62:35567)
0927 15:07:36.807772 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:36.807779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:36.807988 (+ 209us) server_negotiation.cc:407] Connection header received
0927 15:07:36.808115 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:36.808119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:36.808243 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:36.808397 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:36.809222 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.809968 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.811351 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.811787 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.811936 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.816053 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40775: BlockingRecv error: recv got EOF from 127.0.97.1:40775 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 27 15:07:37 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:37 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:37.312326 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.303584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:41571)
0927 15:07:37.304111 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:37.304140 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:37.304307 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:37.304662 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:37.304666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:37.305065 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:37.305223 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.305232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.306232 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.306238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.307301 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.307314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.308018 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.308024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.308222 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.308881 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:37.308899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:37.312016 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":262,"threads_started":1}
W20260927 15:07:37.312636 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:37.303839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41571 (local address 127.0.97.62:35567)
0927 15:07:37.304085 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:37.304092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:37.304225 (+ 133us) server_negotiation.cc:407] Connection header received
0927 15:07:37.304464 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:37.304469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:37.304528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:37.304631 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:37.305368 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.306092 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.307442 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.307866 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.308003 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.312328 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41571: BlockingRecv error: recv got EOF from 127.0.97.2:41571 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 27 15:07:37 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:37 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:37.790198 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.781986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:58257)
0927 15:07:37.782473 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:37.782489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:37.782608 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:37.782960 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:37.782963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:37.783333 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:37.783477 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.783486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.784464 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.784468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.785478 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.785491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.786145 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.786149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.786301 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.786893 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:37.786909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:37.789945 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
W20260927 15:07:37.790473 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:37.782048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58257 (local address 127.0.97.62:35567)
0927 15:07:37.782280 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:37.782290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:37.782573 (+ 283us) server_negotiation.cc:407] Connection header received
0927 15:07:37.782746 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:37.782750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:37.782831 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:37.782916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:37.783612 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.784322 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.785623 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.786024 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.786119 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.790185 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58257: BlockingRecv error: recv got EOF from 127.0.97.3:58257 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 15:07:37 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:37 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:37.825919 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.817152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:48113)
0927 15:07:37.817738 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:37.817761 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:37.817933 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:37.818303 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:37.818307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:37.818855 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:37.819139 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.819152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.820185 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.820193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.821152 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.821165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.821911 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.821918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.822100 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.822743 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:37.822760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:37.825665 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":305,"threads_started":1}
W20260927 15:07:37.826227 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:37.817258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48113 (local address 127.0.97.62:35567)
0927 15:07:37.817499 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:37.817505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:37.817935 (+ 430us) server_negotiation.cc:407] Connection header received
0927 15:07:37.818075 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:37.818081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:37.818153 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:37.818269 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:37.819309 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.820036 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.821321 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.821736 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.821893 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.825959 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48113: BlockingRecv error: recv got EOF from 127.0.97.1:48113 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 15:07:38 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:38 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:38.322327 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.313398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:51315)
0927 15:07:38.313971 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:38.313987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:38.314099 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:38.314424 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:38.314428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:38.314859 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:38.315017 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.315026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.316180 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.316184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.317138 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.317151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.317876 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.317881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.318006 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.318797 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:38.318816 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:38.321996 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":304,"threads_started":1}
W20260927 15:07:38.322487 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:38.313583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51315 (local address 127.0.97.62:35567)
0927 15:07:38.313819 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:38.313824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:38.314111 (+ 287us) server_negotiation.cc:407] Connection header received
0927 15:07:38.314196 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:38.314201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:38.314273 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:38.314372 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:38.315235 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.316012 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.317298 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.317726 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.317841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.322211 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51315: BlockingRecv error: recv got EOF from 127.0.97.2:51315 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:07:38 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:38 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:38.800068 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.791365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:60339)
0927 15:07:38.791902 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:38.791921 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:38.792042 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:38.792456 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:38.792459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:38.792843 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:38.793003 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.793011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.794002 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.794006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.794911 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.794923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.795657 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.795662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.795793 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.796367 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:38.796393 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:38.799711 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
W20260927 15:07:38.800208 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:38.791501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60339 (local address 127.0.97.62:35567)
0927 15:07:38.791733 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:38.791738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:38.792008 (+ 270us) server_negotiation.cc:407] Connection header received
0927 15:07:38.792198 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:38.792202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:38.792280 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:38.792400 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:38.793133 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.793862 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.795049 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.795509 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.795605 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.799909 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60339: BlockingRecv error: recv got EOF from 127.0.97.3:60339 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 15:07:38 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:38 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:38.835345 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.827019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:40515)
0927 15:07:38.827609 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:38.827625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:38.827741 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:38.828052 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:38.828055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:38.828407 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:38.828559 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.828566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.829589 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.829592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.830584 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.830597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.831385 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.831389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.831510 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.832105 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:38.832124 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:38.835032 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":368,"threads_started":1}
W20260927 15:07:38.835641 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:38.827113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40515 (local address 127.0.97.62:35567)
0927 15:07:38.827302 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:38.827308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:38.827701 (+ 393us) server_negotiation.cc:407] Connection header received
0927 15:07:38.827840 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:38.827844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:38.827920 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:38.828007 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:38.828688 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.829464 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.830732 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.831252 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.831355 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.835379 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40515: BlockingRecv error: recv got EOF from 127.0.97.1:40515 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 15:07:39 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:39 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:39.336576 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.323464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:35255)
0927 15:07:39.324270 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.324287 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.324456 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.324904 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.324909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.325438 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:39.325586 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.325596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.326719 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.326725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.327729 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.327743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.328504 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.328511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.328690 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.333000 (+ 4310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:39.333022 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:39.336219 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":562,"threads_started":1}
W20260927 15:07:39.336727 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:39.323614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35255 (local address 127.0.97.62:35567)
0927 15:07:39.323817 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.323825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.324379 (+ 554us) server_negotiation.cc:407] Connection header received
0927 15:07:39.324652 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.324656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.324721 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.324873 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.325781 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.326557 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.327892 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.328327 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.328465 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.336467 (+ 8002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35255: BlockingRecv error: recv got EOF from 127.0.97.2:35255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:07:39 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:39 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:39.810112 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.801193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:33413)
0927 15:07:39.801778 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.801794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.801973 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.802313 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.802317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.802806 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:39.803000 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.803009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.804252 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.804258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.805173 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.805184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.805960 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.805966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.806138 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.806774 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:39.806792 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:39.809884 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":345,"threads_started":1}
W20260927 15:07:39.810396 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:39.801364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33413 (local address 127.0.97.62:35567)
0927 15:07:39.801653 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.801660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.801897 (+ 237us) server_negotiation.cc:407] Connection header received
0927 15:07:39.802110 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.802114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.802175 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.802283 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.803184 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.804113 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.805337 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.805790 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.805928 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.810111 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33413: BlockingRecv error: recv got EOF from 127.0.97.3:33413 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 15:07:39 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:39 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:39.846091 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.836515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:57539)
0927 15:07:39.837147 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.837163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.837289 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.837621 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.837624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.838034 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:39.838201 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.838210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.839814 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.839818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.841128 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.841146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.842004 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.842009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.842138 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.842734 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:39.842751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:39.845847 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":307,"threads_started":1}
W20260927 15:07:39.846306 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:39.836602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57539 (local address 127.0.97.62:35567)
0927 15:07:39.836837 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.836864 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.837263 (+ 399us) server_negotiation.cc:407] Connection header received
0927 15:07:39.837403 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.837407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.837476 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.837577 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.838393 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.839626 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.841339 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.841820 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.841927 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.846091 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57539: BlockingRecv error: recv got EOF from 127.0.97.1:57539 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 15:07:40 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:40 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:40.347877 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.337823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:53189)
0927 15:07:40.338565 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.338583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.338718 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.339144 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.339148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.339690 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.339895 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.339906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.340996 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.341000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.341964 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.341978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.342701 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.342706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.342846 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.343698 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:40.343724 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.347521 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":308,"threads_started":1}
W20260927 15:07:40.347958 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:40.337934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53189 (local address 127.0.97.62:35567)
0927 15:07:40.338180 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.338186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.338755 (+ 569us) server_negotiation.cc:407] Connection header received
0927 15:07:40.338826 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.338830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.338908 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.339009 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.340068 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.340822 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.342119 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.342523 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.342626 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.347747 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53189: BlockingRecv error: recv got EOF from 127.0.97.2:53189 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 15:07:40 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:40 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:40.820415 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:40.811587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42713 (local address 127.0.97.62:35567)
0927 15:07:40.811836 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.811844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.812342 (+ 498us) server_negotiation.cc:407] Connection header received
0927 15:07:40.812570 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.812574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.812652 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.812765 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.813686 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.814432 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.815815 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.816291 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.816390 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.820207 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42713: BlockingRecv error: recv got EOF from 127.0.97.3:42713 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:07:40.820423 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.811489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:42713)
0927 15:07:40.812247 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.812264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.812430 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.812816 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.812820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.813306 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.813523 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.813535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.814585 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.814589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.815675 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.815688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.816427 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.816432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.816539 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.817114 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:40.817131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.819986 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":362,"threads_started":1}
Sep 27 15:07:40 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:40 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:40.856766 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.847342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:54909)
0927 15:07:40.847994 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.848012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.848140 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.848612 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.848617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.849189 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.849424 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.849436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.850691 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.850698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.851656 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.851669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.852438 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.852444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.852631 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.853262 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:40.853279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.856525 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":295,"threads_started":1}
W20260927 15:07:40.857074 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:40.847645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54909 (local address 127.0.97.62:35567)
0927 15:07:40.847894 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.847901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.848113 (+ 212us) server_negotiation.cc:407] Connection header received
0927 15:07:40.848270 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.848275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.848344 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.848592 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.849588 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.850533 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.851816 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.852237 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.852379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.856786 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54909: BlockingRecv error: recv got EOF from 127.0.97.1:54909 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:07:41 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:41 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:41.358713 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.349056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:44481)
0927 15:07:41.349632 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:41.349650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:41.349788 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:41.350121 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:41.350124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:41.350527 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:41.350675 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.350684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.352249 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.352255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.353505 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.353520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.354260 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.354265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.354524 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.355415 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:41.355433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:41.358477 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":347,"threads_started":1}
W20260927 15:07:41.358980 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:41.349206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44481 (local address 127.0.97.62:35567)
0927 15:07:41.349476 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.349482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.349789 (+ 307us) server_negotiation.cc:407] Connection header received
0927 15:07:41.349902 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.349907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.349975 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.350073 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:41.350842 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.352048 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.353642 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.354090 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.354191 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.358714 (+ 4523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44481: BlockingRecv error: recv got EOF from 127.0.97.2:44481 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:07:41 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:41 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:41.830607 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.821642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:60277)
0927 15:07:41.822153 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:41.822170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:41.822338 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:41.822735 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:41.822739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:41.823327 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:41.823490 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.823499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.824534 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.824541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.825478 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.825491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.826270 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.826276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.826462 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.827186 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:41.827204 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:41.830255 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":270,"threads_started":1}
W20260927 15:07:41.830754 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:41.821750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60277 (local address 127.0.97.62:35567)
0927 15:07:41.822027 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.822033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.822277 (+ 244us) server_negotiation.cc:407] Connection header received
0927 15:07:41.822486 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.822492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.822586 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.822702 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:41.823643 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.824364 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.825639 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.826083 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.826225 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.830499 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60277: BlockingRecv error: recv got EOF from 127.0.97.3:60277 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 15:07:41 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:41 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:41.867969 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.857912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:56953)
0927 15:07:41.858482 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:41.858500 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:41.858630 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:41.858947 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:41.858951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:41.859478 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:41.859690 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.859703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.861123 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.861127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.862717 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.862732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.863672 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.863678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.863857 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.864619 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:41.864644 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:41.867702 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":328,"threads_started":1}
W20260927 15:07:41.868386 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:41.858019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56953 (local address 127.0.97.62:35567)
0927 15:07:41.858291 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.858299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.858595 (+ 296us) server_negotiation.cc:407] Connection header received
0927 15:07:41.858736 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.858740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.858804 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.858898 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:41.859887 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.860937 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.862861 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.863507 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.863652 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.868009 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56953: BlockingRecv error: recv got EOF from 127.0.97.1:56953 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 15:07:42 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:42 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:42.370134 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:42.359771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:35847)
0927 15:07:42.360349 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:42.360404 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:42.360534 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:42.360897 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:42.360901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:42.361355 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:42.361572 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:42.361582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.362790 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.362794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:42.364091 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:42.364105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.365378 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.365385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:42.365564 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:42.366257 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:42.366283 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:42.369761 (+ 3478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":322,"threads_started":1}
W20260927 15:07:42.370261 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:42.359864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35847 (local address 127.0.97.62:35567)
0927 15:07:42.360159 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:42.360166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:42.360524 (+ 358us) server_negotiation.cc:407] Connection header received
0927 15:07:42.360654 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:42.360657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:42.360723 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:42.360817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:42.361747 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.362643 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.364240 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.365122 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.365280 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:42.370017 (+ 4737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35847: BlockingRecv error: recv got EOF from 127.0.97.2:35847 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 15:07:42 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:42 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:42.840786 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:42.831707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:60659)
0927 15:07:42.832231 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:42.832249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:42.832404 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:42.832768 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:42.832772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:42.833164 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:42.833381 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:42.833390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.834550 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.834555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:42.835720 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:42.835734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.836453 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.836457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:42.836584 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:42.837294 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:42.837313 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:42.840394 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":296,"threads_started":1}
W20260927 15:07:42.840915 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:42.831756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60659 (local address 127.0.97.62:35567)
0927 15:07:42.832008 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:42.832018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:42.832346 (+ 328us) server_negotiation.cc:407] Connection header received
0927 15:07:42.832523 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:42.832527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:42.832600 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:42.832686 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:42.833567 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.834402 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.835866 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.836290 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.836390 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:42.840644 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60659: BlockingRecv error: recv got EOF from 127.0.97.3:60659 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 15:07:42 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:42 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:42.877758 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:42.869031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:59529)
0927 15:07:42.869541 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:42.869559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:42.869682 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:42.869980 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:42.869983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:42.870357 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:42.870531 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:42.870540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.871703 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.871708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:42.872810 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:42.872823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.873540 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.873546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:42.873677 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:42.874433 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:42.874451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:42.877393 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":289,"threads_started":1}
W20260927 15:07:42.878063 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:42.869101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59529 (local address 127.0.97.62:35567)
0927 15:07:42.869343 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:42.869353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:42.869661 (+ 308us) server_negotiation.cc:407] Connection header received
0927 15:07:42.869789 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:42.869792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:42.869855 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:42.869935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:42.870699 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.871557 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.872971 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:42.873372 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:42.873467 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:42.877598 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59529: BlockingRecv error: recv got EOF from 127.0.97.1:59529 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 15:07:43 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:43 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:43.381678 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.371170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:56083)
0927 15:07:43.371844 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:43.371861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:43.372029 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:43.372433 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:43.372437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:43.372836 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:43.372975 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.372983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.374105 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.374115 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.375705 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.375720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.376594 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.376600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.376793 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.377601 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:43.377620 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:43.381325 (+ 3705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":443,"threads_started":1}
W20260927 15:07:43.381821 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:43.371437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56083 (local address 127.0.97.62:35567)
0927 15:07:43.371736 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.371743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.371948 (+ 205us) server_negotiation.cc:407] Connection header received
0927 15:07:43.372183 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.372187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.372248 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.372415 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.373120 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.373945 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.375878 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.376408 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.376564 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.381596 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56083: BlockingRecv error: recv got EOF from 127.0.97.2:56083 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 27 15:07:43 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:43 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:43.851147 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.842128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:33177)
0927 15:07:43.842677 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:43.842693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:43.842815 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:43.843209 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:43.843212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:43.843579 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:43.843721 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.843730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.844762 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.844765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.845712 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.845725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.846517 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.846521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.846648 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.847412 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:43.847432 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:43.850841 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":318,"threads_started":1}
W20260927 15:07:43.851383 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:43.842343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33177 (local address 127.0.97.62:35567)
0927 15:07:43.842540 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.842545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.842811 (+ 266us) server_negotiation.cc:407] Connection header received
0927 15:07:43.842910 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.842914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.842996 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.843179 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.843878 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.844625 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.845868 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.846363 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.846475 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.851051 (+ 4576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33177: BlockingRecv error: recv got EOF from 127.0.97.3:33177 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 15:07:43 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:43 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:43.888332 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.878690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:35307)
0927 15:07:43.879266 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:43.879281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:43.879449 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:43.879829 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:43.879834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:43.880375 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:43.880557 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.880566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.881615 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.881623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.882582 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.882594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.883459 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.883466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.883653 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.884294 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:43.884313 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:43.887951 (+ 3638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":313,"threads_started":1}
W20260927 15:07:43.888487 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:43.878816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35307 (local address 127.0.97.62:35567)
0927 15:07:43.879112 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.879120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.879371 (+ 251us) server_negotiation.cc:407] Connection header received
0927 15:07:43.879576 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.879588 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.879664 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.879776 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.880710 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.881474 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.882744 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.883279 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.883424 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.888191 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35307: BlockingRecv error: recv got EOF from 127.0.97.1:35307 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 15:07:44 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:44 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:44.397717 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:44.382718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:34387)
0927 15:07:44.383331 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:44.383348 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:44.383511 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:44.383861 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:44.383866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:44.384300 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:44.384472 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:44.384483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.385577 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.385584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:44.386495 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:44.386508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.387313 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.387319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:44.387493 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:44.393813 (+ 6320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:44.393836 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:44.397356 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":356,"threads_started":1}
W20260927 15:07:44.397950 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:44.382827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34387 (local address 127.0.97.62:35567)
0927 15:07:44.383114 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:44.383123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:44.383440 (+ 317us) server_negotiation.cc:407] Connection header received
0927 15:07:44.383647 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:44.383659 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:44.383721 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:44.383830 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:44.384626 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.385385 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.386646 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.387133 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.387267 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:44.397679 (+ 10412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34387: BlockingRecv error: recv got EOF from 127.0.97.2:34387 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 15:07:44 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:44 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:44.861399 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:44.852268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:59177)
0927 15:07:44.852828 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:44.852845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:44.852994 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:44.853443 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:44.853447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:44.853850 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:44.854018 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:44.854026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.855113 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.855117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:44.856049 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:44.856062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.856808 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.856814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:44.857005 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:44.857815 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:44.857843 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:44.861092 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":302,"threads_started":1}
W20260927 15:07:44.861546 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:44.852407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59177 (local address 127.0.97.62:35567)
0927 15:07:44.852648 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:44.852656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:44.852958 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:07:44.853144 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:44.853149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:44.853236 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:44.853336 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:44.854166 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.854919 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.856205 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.856611 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.856779 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:44.861320 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59177: BlockingRecv error: recv got EOF from 127.0.97.3:59177 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:07:44 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:44 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:44.898784 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:44.889506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:59835)
0927 15:07:44.890058 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:44.890073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:44.890236 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:44.890603 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:44.890610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:44.891035 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:44.891267 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:44.891276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.892309 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.892315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:44.893221 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:44.893232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.893962 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.893967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:44.894160 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:44.894944 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:44.894974 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:44.898554 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":330,"threads_started":1}
W20260927 15:07:44.899011 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:44.889593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59835 (local address 127.0.97.62:35567)
0927 15:07:44.889798 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:44.889805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:44.890169 (+ 364us) server_negotiation.cc:407] Connection header received
0927 15:07:44.890370 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:44.890379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:44.890443 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:44.890555 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:44.891422 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.892165 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.893367 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:44.893806 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:44.893936 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:44.898739 (+ 4803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59835: BlockingRecv error: recv got EOF from 127.0.97.1:59835 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 15:07:45 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:45 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:45.407280 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.398784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:36597)
0927 15:07:45.399407 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:45.399432 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:45.399553 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:45.399878 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:45.399881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:45.400277 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:45.400430 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.400439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.401464 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.401469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.402388 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.402400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.403209 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.403214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.403368 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.404042 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:45.404061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:45.406967 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":346,"threads_started":1}
W20260927 15:07:45.407495 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:45.398917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36597 (local address 127.0.97.62:35567)
0927 15:07:45.399236 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.399242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.399519 (+ 277us) server_negotiation.cc:407] Connection header received
0927 15:07:45.399670 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.399674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.399735 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.399819 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.400566 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.401304 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.402539 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.402962 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.403160 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.407273 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36597: BlockingRecv error: recv got EOF from 127.0.97.2:36597 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 27 15:07:45 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:45 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:45.871819 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.862465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:60527)
0927 15:07:45.863123 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:45.863145 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:45.863289 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:45.863734 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:45.863738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:45.864227 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:45.864384 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.864392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.865364 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.865369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.866287 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.866298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.867364 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.867369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.867544 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.868427 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:45.868446 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:45.871556 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":295,"threads_started":1}
W20260927 15:07:45.872108 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:45.862497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60527 (local address 127.0.97.62:35567)
0927 15:07:45.862659 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.862663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.863234 (+ 571us) server_negotiation.cc:407] Connection header received
0927 15:07:45.863434 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.863440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.863578 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.863699 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.864523 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.865224 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.866481 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.867188 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.867296 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.871788 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60527: BlockingRecv error: recv got EOF from 127.0.97.3:60527 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 15:07:45 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:45 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:45.908811 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.899822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:49505)
0927 15:07:45.900349 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:45.900369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:45.900559 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:45.900913 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:45.900918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:45.901380 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:45.901550 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.901559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.902578 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.902583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.903766 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.903779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.904559 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.904566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.904732 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.905366 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:45.905383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:45.908459 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":422,"threads_started":1}
W20260927 15:07:45.908907 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:45.899912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49505 (local address 127.0.97.62:35567)
0927 15:07:45.900120 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.900129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.900472 (+ 343us) server_negotiation.cc:407] Connection header received
0927 15:07:45.900693 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.900699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.900771 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.900876 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.901696 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.902426 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.903924 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.904358 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.904503 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.908673 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49505: BlockingRecv error: recv got EOF from 127.0.97.1:49505 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:07:46 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:46 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:46.417733 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.408434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:39725)
0927 15:07:46.409320 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.409407 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.409574 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.410016 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.410022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.410521 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.410709 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.410719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.411949 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.411954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.412973 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.412987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.413734 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.413739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.413866 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.414580 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.414599 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.417486 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":508,"threads_started":1}
W20260927 15:07:46.418082 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:46.408719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39725 (local address 127.0.97.62:35567)
0927 15:07:46.408978 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.408985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.409521 (+ 536us) server_negotiation.cc:407] Connection header received
0927 15:07:46.409732 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.409738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.409826 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.409939 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.410851 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.411774 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.413115 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.413531 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.413632 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.417760 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39725: BlockingRecv error: recv got EOF from 127.0.97.2:39725 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 27 15:07:46 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:46 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:46.882405 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.872903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:58587)
0927 15:07:46.873476 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.873496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.873643 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.874022 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.874026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.874446 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.874609 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.874619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.875901 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.875907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.877376 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.877397 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.878432 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.878437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.878557 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.879231 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.879261 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.882170 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
W20260927 15:07:46.882673 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:46.873021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58587 (local address 127.0.97.62:35567)
0927 15:07:46.873307 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.873313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.873608 (+ 295us) server_negotiation.cc:407] Connection header received
0927 15:07:46.873754 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.873758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.873841 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.873961 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.874768 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.875657 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.877576 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.878201 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.878325 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.882425 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58587: BlockingRecv error: recv got EOF from 127.0.97.3:58587 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 27 15:07:46 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:46 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:46.919060 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.910025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:36737)
0927 15:07:46.910561 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.910579 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.910699 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.911142 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.911149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.911552 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.911731 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.911739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.913012 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.913016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.913966 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.913978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.914893 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.914899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.915154 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.915967 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.915994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.918806 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":298,"threads_started":1}
W20260927 15:07:46.919303 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:46.910149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36737 (local address 127.0.97.62:35567)
0927 15:07:46.910568 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.910573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.910659 (+ 86us) server_negotiation.cc:407] Connection header received
0927 15:07:46.910804 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.910808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.910897 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.911040 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.911902 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.912862 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.914121 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.914717 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.914871 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.919021 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36737: BlockingRecv error: recv got EOF from 127.0.97.1:36737 (error 108)
Metrics: {"server-negotiator.queue_time_us":281}
Sep 27 15:07:47 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:47 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:47.427464 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.418781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:56017)
0927 15:07:47.419286 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.419303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.419471 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.419848 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.419851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.420220 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.420391 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.420401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.421458 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.421461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.422393 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.422404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.423224 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.423228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.423349 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.424059 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.424078 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.427210 (+ 3132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
W20260927 15:07:47.427704 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:47.418980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56017 (local address 127.0.97.62:35567)
0927 15:07:47.419230 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.419237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.419409 (+ 172us) server_negotiation.cc:407] Connection header received
0927 15:07:47.419651 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.419654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.419715 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.419798 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.420556 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.421283 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.422554 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.423014 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.423218 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.427481 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56017: BlockingRecv error: recv got EOF from 127.0.97.2:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 27 15:07:47 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:47 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:47.893033 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.883447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:59855)
0927 15:07:47.883923 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.883942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.884065 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.884422 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.884426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.884826 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.885022 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.885032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.886268 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.886277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.887274 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.887296 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.888287 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.888294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.888438 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.889299 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.889321 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.892555 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":218,"threads_started":1}
W20260927 15:07:47.893092 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:47.883669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59855 (local address 127.0.97.62:35567)
0927 15:07:47.883905 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.883914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.884048 (+ 134us) server_negotiation.cc:407] Connection header received
0927 15:07:47.884170 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.884174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.884240 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.884349 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.885203 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.886057 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.887646 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.888085 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.888217 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.892866 (+ 4649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59855: BlockingRecv error: recv got EOF from 127.0.97.3:59855 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 15:07:47 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:47 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:47.928900 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.920445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:39219)
0927 15:07:47.920938 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.920953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.921111 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.921476 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.921481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.921933 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.922119 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.922128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.923205 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.923213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.924141 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.924152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.924895 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.924901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.925072 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.925667 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.925684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.928642 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":252,"threads_started":1}
W20260927 15:07:47.929389 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:47.920454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39219 (local address 127.0.97.62:35567)
0927 15:07:47.920684 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.920692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.921041 (+ 349us) server_negotiation.cc:407] Connection header received
0927 15:07:47.921249 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.921254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.921338 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.921477 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.922259 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.923029 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.924294 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.924740 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.924879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.928909 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39219: BlockingRecv error: recv got EOF from 127.0.97.1:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 15:07:48 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:48 dist-test-slave-g0w1 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:48.438651 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.428722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:39623)
0927 15:07:48.429276 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.429307 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:48.429507 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:48.429924 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:48.429928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:48.430380 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:48.430566 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:48.430575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.431928 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.431935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:48.433003 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:48.433018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.433930 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.433937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:48.434141 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.434859 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:48.434880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:48.438275 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35567: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":331,"threads_started":1}
W20260927 15:07:48.438745 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:48.428891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39623 (local address 127.0.97.62:35567)
0927 15:07:48.429065 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:48.429070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:48.429440 (+ 370us) server_negotiation.cc:407] Connection header received
0927 15:07:48.429706 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:48.429720 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:48.429782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:48.429891 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:48.430720 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.431561 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.433184 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.433747 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.433919 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.438502 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39623: BlockingRecv error: recv got EOF from 127.0.97.2:39623 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:07:48.722014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9231
I20260927 15:07:48.735838 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.723598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.0.97.1:42529)
0927 15:07:48.724186 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:48.724194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:48.724222 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:07:48.724282 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:48.724289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:48.724528 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:48.724691 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:48.726244 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.727434 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.728879 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.729518 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.729628 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.732310 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:48.732319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:48.732334 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:48.732429 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:48.735315 (+ 2886us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:07:48.735599 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49994: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":271,"threads_started":1}
W20260927 15:07:48.735977 9691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49994: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:07:48.736276 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9231: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42529: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:07:48.736474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9231
I20260927 15:07:48.753340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9382
I20260927 15:07:48.766975 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.754450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.0.97.2:46777)
0927 15:07:48.755038 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:48.755046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:48.755135 (+ 89us) server_negotiation.cc:407] Connection header received
0927 15:07:48.755236 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:48.755243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:48.755495 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:48.755734 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:48.757227 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.758409 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.759983 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.760616 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.760784 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.763697 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:48.763708 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:48.763722 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:48.763767 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:48.766233 (+ 2466us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:07:48.766620 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45986: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":270,"threads_started":1}
W20260927 15:07:48.767128 9692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45986: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:07:48.767174 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9382: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46777: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:07:48.767355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9382
I20260927 15:07:48.784650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9513
I20260927 15:07:48.796967 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.785663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.97.3:43489)
0927 15:07:48.786175 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:48.786182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:48.786209 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:48.786268 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:48.786274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:48.786488 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:48.786691 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:48.788108 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.789227 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.790440 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.791030 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.791265 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.793707 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:48.793715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:48.793730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:48.793767 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:48.796425 (+ 2658us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:07:48.796702 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51422: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":295,"threads_started":1}
W20260927 15:07:48.797079 9693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51422: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:07:48.797175 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9513: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:43489: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:07:48.797331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9513
I20260927 15:07:48.813649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9162
I20260927 15:07:48.814975 9224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:48.929630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9162
2026-09-27T15:07:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21415 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T15:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:48Z Disabled control of system clock
I20260927 15:07:49.009146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40645
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:07:49.347981 9698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:49.348351 9698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:49.348433 9698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:49.355821 9698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:49.355947 9698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:49.355984 9698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:49.356033 9698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:49.356077 9698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:49.365348 9698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:49.366654 9698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:49.368418 9698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:49.380744 9703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:49.380697 9706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:49.380697 9704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:49.381448 9698 server_base.cc:1034] running on GCE node
I20260927 15:07:49.382315 9698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:49.383862 9698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:49.385108 9698 hybrid_clock.cc:648] HybridClock initialized: now 1790521669385070 us; error 53 us; skew 500 ppm
I20260927 15:07:49.385501 9698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:49.387928 9698 webserver.cc:460] Webserver started at http://127.0.97.62:32853/ using document root <none> and password file <none>
I20260927 15:07:49.388665 9698 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:49.388799 9698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:49.389130 9698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:49.392426 9698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "aeb46cabcf644e05a8f811e41f823edf"
format_stamp: "Formatted at 2026-09-27 15:07:49 on dist-test-slave-g0w1"
server_key: "1d5fa370a856f74d832eecf69565fa7b"
server_key_iv: "d56c8fb1f1eede117e78af3c97a06062"
server_key_version: "encryptionkey@0"
I20260927 15:07:49.393275 9698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "aeb46cabcf644e05a8f811e41f823edf"
format_stamp: "Formatted at 2026-09-27 15:07:49 on dist-test-slave-g0w1"
server_key: "1d5fa370a856f74d832eecf69565fa7b"
server_key_iv: "d56c8fb1f1eede117e78af3c97a06062"
server_key_version: "encryptionkey@0"
I20260927 15:07:49.398823 9698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:07:49.402849 9712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:49.404613 9698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:49.404852 9698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "aeb46cabcf644e05a8f811e41f823edf"
format_stamp: "Formatted at 2026-09-27 15:07:49 on dist-test-slave-g0w1"
server_key: "1d5fa370a856f74d832eecf69565fa7b"
server_key_iv: "d56c8fb1f1eede117e78af3c97a06062"
server_key_version: "encryptionkey@0"
I20260927 15:07:49.405114 9698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:49.430321 9698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:49.431322 9698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:49.431636 9698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:49.450876 9698 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40645
I20260927 15:07:49.450930 9763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40645 every 8 connection(s)
I20260927 15:07:49.452380 9698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:49.456764 9764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:49.461522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9698
I20260927 15:07:49.461769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:49.462430 388 external_mini_cluster.cc:1428] Setting key 3775895a827cdd67a904c6dcbf4fd051
I20260927 15:07:49.468348 9764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf: Bootstrap starting.
I20260927 15:07:49.471640 9764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:49.472728 9764 log.cc:824] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:49.475898 9764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf: No bootstrap required, opened a new log
I20260927 15:07:49.476805 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:49.464656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.0.97.62:40645)
0927 15:07:49.465792 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:49.465804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:49.465859 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:07:49.467690 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:49.467716 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:49.468245 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:49.468713 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:49.469336 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:49.470649 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:49.472193 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:49.472959 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:49.473184 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:49.474378 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:49.474386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:49.474402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:49.474952 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:49.475039 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:49.475064 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:49.475253 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:49.475477 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:07:49.475807 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":423,"threads_started":1}
I20260927 15:07:49.483821 9764 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb46cabcf644e05a8f811e41f823edf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40645 } }
I20260927 15:07:49.484246 9764 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:49.484365 9764 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb46cabcf644e05a8f811e41f823edf, State: Initialized, Role: FOLLOWER
I20260927 15:07:49.484894 9764 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb46cabcf644e05a8f811e41f823edf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40645 } }
I20260927 15:07:49.485085 9764 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:49.485177 9764 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:49.485301 9764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:49.487551 9764 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb46cabcf644e05a8f811e41f823edf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40645 } }
I20260927 15:07:49.488006 9764 leader_election.cc:302] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeb46cabcf644e05a8f811e41f823edf; no voters:
I20260927 15:07:49.488572 9764 leader_election.cc:288] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:49.488695 9769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:49.489866 9769 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [term 1 LEADER]: Becoming Leader. State: Replica: aeb46cabcf644e05a8f811e41f823edf, State: Running, Role: LEADER
I20260927 15:07:49.490433 9769 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb46cabcf644e05a8f811e41f823edf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40645 } }
I20260927 15:07:49.491143 9764 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:49.493737 9770 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeb46cabcf644e05a8f811e41f823edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb46cabcf644e05a8f811e41f823edf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40645 } } }
I20260927 15:07:49.494223 9770 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:49.498765 9771 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb46cabcf644e05a8f811e41f823edf. Latest consensus state: current_term: 1 leader_uuid: "aeb46cabcf644e05a8f811e41f823edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb46cabcf644e05a8f811e41f823edf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40645 } } }
I20260927 15:07:49.499207 9771 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:49.499951 9778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:49.506001 9778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:49.516237 9778 catalog_manager.cc:1347] Generated new cluster ID: 07e54f22f4c74a04b4601fbcb2894ec1
I20260927 15:07:49.516539 9778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:49.525475 9778 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:49.526859 9778 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:49.544386 9778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeb46cabcf644e05a8f811e41f823edf: Generated new TSK 0
I20260927 15:07:49.545606 9778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:49.561820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40645
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 15:07:49.902760 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:49.903053 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:49.903264 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:49.910928 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:49.911106 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:49.911288 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:49.920717 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:49.922068 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:49.923959 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:49.937882 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:49.937882 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:49.937940 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:49.938709 9788 server_base.cc:1034] running on GCE node
I20260927 15:07:49.939461 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:49.940491 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:49.941730 9788 hybrid_clock.cc:648] HybridClock initialized: now 1790521669941669 us; error 48 us; skew 500 ppm
I20260927 15:07:49.942054 9788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:49.944591 9788 webserver.cc:460] Webserver started at http://127.0.97.1:44289/ using document root <none> and password file <none>
I20260927 15:07:49.945309 9788 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:49.945397 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:49.945670 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:49.948624 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "7a5ab7583d814d859db19ff3c79a71a0"
format_stamp: "Formatted at 2026-09-27 15:07:49 on dist-test-slave-g0w1"
server_key: "bb6bca3318ba78fd4db6ae7706982f0c"
server_key_iv: "b18774d6ce3074306a2de1a215aea1f1"
server_key_version: "encryptionkey@0"
I20260927 15:07:49.949414 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7a5ab7583d814d859db19ff3c79a71a0"
format_stamp: "Formatted at 2026-09-27 15:07:49 on dist-test-slave-g0w1"
server_key: "bb6bca3318ba78fd4db6ae7706982f0c"
server_key_iv: "b18774d6ce3074306a2de1a215aea1f1"
server_key_version: "encryptionkey@0"
I20260927 15:07:49.954833 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:07:49.960386 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:49.962275 9788 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:49.962482 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "7a5ab7583d814d859db19ff3c79a71a0"
format_stamp: "Formatted at 2026-09-27 15:07:49 on dist-test-slave-g0w1"
server_key: "bb6bca3318ba78fd4db6ae7706982f0c"
server_key_iv: "b18774d6ce3074306a2de1a215aea1f1"
server_key_version: "encryptionkey@0"
I20260927 15:07:49.962762 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:49.981441 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:49.982374 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:49.982622 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:49.983680 9788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:49.985178 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:49.985273 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:49.985369 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:49.985422 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.019618 9788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45927
I20260927 15:07:50.019675 9914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45927 every 8 connection(s)
I20260927 15:07:50.020998 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:50.025274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9788
I20260927 15:07:50.025490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:50.026110 388 external_mini_cluster.cc:1428] Setting key 9141e019329052d7679c845d2cb20526
I20260927 15:07:50.028766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40645
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:07:50.034374 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.023671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34973 (local address 127.0.97.62:40645)
0927 15:07:50.024138 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.024144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.025502 (+ 1358us) server_negotiation.cc:407] Connection header received
0927 15:07:50.027385 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.027391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.027493 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.027615 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:50.028759 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.029598 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.031367 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.031786 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.031929 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.033474 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:50.033480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:50.033487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:50.033513 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:50.033555 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:50.033566 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:50.033665 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.033904 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.034016 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260927 15:07:50.035574 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.024340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40645 (local address 127.0.97.1:34973)
0927 15:07:50.025271 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.025316 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.027061 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.027866 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.027875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.027976 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:50.028586 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.028603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.029779 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.029783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.031198 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.031217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.032052 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.032059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.032325 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.033123 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:50.033126 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:50.033171 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:50.033258 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:50.033665 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:50.033679 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:50.033690 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.033893 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.034342 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":278,"threads_started":1}
I20260927 15:07:50.037312 9915 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40645
I20260927 15:07:50.037683 9915 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:50.038664 9915 heartbeater.cc:511] Master 127.0.97.62:40645 requested a full tablet report, sending...
I20260927 15:07:50.041395 9729 ts_manager.cc:194] Registered new tserver with Master: 7a5ab7583d814d859db19ff3c79a71a0 (127.0.97.1:45927)
I20260927 15:07:50.043759 9729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34973
W20260927 15:07:50.375201 9919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:50.375520 9919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:50.375605 9919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:50.383256 9919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:50.383368 9919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:50.383484 9919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:50.393673 9919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:50.395008 9919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:50.396937 9919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:50.410728 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:50.410760 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:50.411346 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:50.411636 9919 server_base.cc:1034] running on GCE node
I20260927 15:07:50.412273 9919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:50.413367 9919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:50.414633 9919 hybrid_clock.cc:648] HybridClock initialized: now 1790521670414569 us; error 71 us; skew 500 ppm
I20260927 15:07:50.414983 9919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:50.417726 9919 webserver.cc:460] Webserver started at http://127.0.97.2:34727/ using document root <none> and password file <none>
I20260927 15:07:50.418570 9919 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:50.418720 9919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:50.419152 9919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:50.422362 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "e809ad7d274c410dafe77ae112ed4628"
format_stamp: "Formatted at 2026-09-27 15:07:50 on dist-test-slave-g0w1"
server_key: "3e478671aa6834f478557666545e7092"
server_key_iv: "d9bdc279fb386c41b44c994156f68a92"
server_key_version: "encryptionkey@0"
I20260927 15:07:50.423367 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e809ad7d274c410dafe77ae112ed4628"
format_stamp: "Formatted at 2026-09-27 15:07:50 on dist-test-slave-g0w1"
server_key: "3e478671aa6834f478557666545e7092"
server_key_iv: "d9bdc279fb386c41b44c994156f68a92"
server_key_version: "encryptionkey@0"
I20260927 15:07:50.429031 9919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:50.432729 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.434253 9919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:50.434397 9919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "e809ad7d274c410dafe77ae112ed4628"
format_stamp: "Formatted at 2026-09-27 15:07:50 on dist-test-slave-g0w1"
server_key: "3e478671aa6834f478557666545e7092"
server_key_iv: "d9bdc279fb386c41b44c994156f68a92"
server_key_version: "encryptionkey@0"
I20260927 15:07:50.434587 9919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:50.451469 9919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:50.452404 9919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:50.452705 9919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:50.453809 9919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:50.455495 9919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:50.455590 9919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.455746 9919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:50.455813 9919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.490782 9919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43499
I20260927 15:07:50.490837 10045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43499 every 8 connection(s)
I20260927 15:07:50.492368 9919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:50.495057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 9919
I20260927 15:07:50.495335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:50.495836 388 external_mini_cluster.cc:1428] Setting key 146dac5b80421ede527f5c4c7e745ab8
I20260927 15:07:50.498672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40645
--builtin_ntp_servers=127.0.97.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:07:50.507470 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.495400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53335 (local address 127.0.97.62:40645)
0927 15:07:50.495624 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.495630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.497499 (+ 1869us) server_negotiation.cc:407] Connection header received
0927 15:07:50.499540 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.499549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.499649 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.499772 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:50.501206 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.502331 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.504134 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.504886 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.505036 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.506593 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:50.506599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:50.506602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:50.506623 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:50.506663 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:50.506675 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:50.506780 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.507032 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.507159 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:07:50.508663 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.496104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40645 (local address 127.0.97.2:53335)
0927 15:07:50.497288 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.497359 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.499226 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.500057 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.500067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.500169 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:50.500998 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.501019 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.502508 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.502512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.503968 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.503981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.505141 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.505150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.505380 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.506170 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:50.506173 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:50.506217 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:50.506307 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:50.506781 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:50.506795 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:50.506806 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.507026 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.507510 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":398,"threads_started":1}
I20260927 15:07:50.510439 10046 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40645
I20260927 15:07:50.510787 10046 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:50.511819 10046 heartbeater.cc:511] Master 127.0.97.62:40645 requested a full tablet report, sending...
I20260927 15:07:50.513489 9729 ts_manager.cc:194] Registered new tserver with Master: e809ad7d274c410dafe77ae112ed4628 (127.0.97.2:43499)
I20260927 15:07:50.514354 9729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53335
W20260927 15:07:50.839864 10049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:50.840178 10049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:50.840323 10049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:50.847838 10049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:50.847958 10049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:50.848140 10049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:50.857703 10049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:50.859134 10049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:50.860956 10049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:50.875231 10055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:50.875272 10057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:50.875190 10054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:50.876132 10049 server_base.cc:1034] running on GCE node
I20260927 15:07:50.876772 10049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:50.877837 10049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:50.879125 10049 hybrid_clock.cc:648] HybridClock initialized: now 1790521670879040 us; error 49 us; skew 500 ppm
I20260927 15:07:50.879477 10049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:50.882145 10049 webserver.cc:460] Webserver started at http://127.0.97.3:33881/ using document root <none> and password file <none>
I20260927 15:07:50.882930 10049 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:50.883126 10049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:50.883481 10049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:50.886544 10049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "bdd07296771a49549724604f74960405"
format_stamp: "Formatted at 2026-09-27 15:07:50 on dist-test-slave-g0w1"
server_key: "a1082852ae4c899c85ea283befa0b74d"
server_key_iv: "248329f9c9fb77b9b26d6bb160c9ba89"
server_key_version: "encryptionkey@0"
I20260927 15:07:50.887485 10049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bdd07296771a49549724604f74960405"
format_stamp: "Formatted at 2026-09-27 15:07:50 on dist-test-slave-g0w1"
server_key: "a1082852ae4c899c85ea283befa0b74d"
server_key_iv: "248329f9c9fb77b9b26d6bb160c9ba89"
server_key_version: "encryptionkey@0"
I20260927 15:07:50.893304 10049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 15:07:50.897710 10063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.899585 10049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:07:50.899814 10049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "bdd07296771a49549724604f74960405"
format_stamp: "Formatted at 2026-09-27 15:07:50 on dist-test-slave-g0w1"
server_key: "a1082852ae4c899c85ea283befa0b74d"
server_key_iv: "248329f9c9fb77b9b26d6bb160c9ba89"
server_key_version: "encryptionkey@0"
I20260927 15:07:50.900110 10049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:50.926229 10049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:50.927446 10049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:50.927788 10049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:50.928879 10049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:50.930549 10049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:50.930649 10049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.930730 10049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:50.930792 10049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:50.967913 10049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40469
I20260927 15:07:50.968020 10175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40469 every 8 connection(s)
I20260927 15:07:50.969368 10049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:50.975240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10049
I20260927 15:07:50.975469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:50.976037 388 external_mini_cluster.cc:1428] Setting key 8b2202788466a3b6afc00211c58a9d67
I20260927 15:07:50.986996 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.972261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51537 (local address 127.0.97.62:40645)
0927 15:07:50.972501 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.972505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.974052 (+ 1547us) server_negotiation.cc:407] Connection header received
0927 15:07:50.976802 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.976809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.976922 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.977066 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:50.978623 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.979606 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.982992 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.983580 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.983704 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.985509 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:50.985514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:50.985518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:50.985541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:50.985575 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:50.985588 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:50.985687 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.986519 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.986638 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 15:07:50.989022 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.972814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40645 (local address 127.0.97.3:51537)
0927 15:07:50.973851 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.973900 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.976405 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.977337 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.977348 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.977465 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:50.978357 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.978381 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.979861 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.979870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.982717 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.982738 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.983866 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.983872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.984136 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.985051 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:50.985054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:50.985103 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:50.985202 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:50.986113 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:50.986125 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:50.986133 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.986340 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.986837 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":341,"threads_started":1}
I20260927 15:07:50.991544 10176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40645
I20260927 15:07:50.992079 10176 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:50.993003 10176 heartbeater.cc:511] Master 127.0.97.62:40645 requested a full tablet report, sending...
I20260927 15:07:50.994989 9729 ts_manager.cc:194] Registered new tserver with Master: bdd07296771a49549724604f74960405 (127.0.97.3:40469)
I20260927 15:07:50.996223 9729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51537
I20260927 15:07:51.006486 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:07:51.013242 9917 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:51.011542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.0.97.62:40645)
0927 15:07:51.011796 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:51.011803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:51.012106 (+ 303us) server_negotiation.cc:407] Connection header received
0927 15:07:51.012234 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:51.012238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:51.012373 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:51.012485 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:51.013004 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40106: BlockingRecv error: recv got EOF from 127.0.0.1:40106 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:07:51.013883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9788
I20260927 15:07:51.022658 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:51.014829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.97.1:45927)
0927 15:07:51.015386 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:51.015394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:51.015421 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:51.015476 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:51.015482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:51.015707 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:51.015835 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:51.016250 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:51.017257 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:51.018578 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:51.019431 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:51.019597 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:51.020844 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:51.020855 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:51.020872 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:51.021639 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:51.021704 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:51.021732 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:51.021880 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:51.022072 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 15:07:51.022215 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":312,"threads_started":1}
I20260927 15:07:51.024183 9910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:51.108506 9915 heartbeater.cc:503] Master 127.0.97.62:40645 was elected leader, sending a full tablet report...
I20260927 15:07:51.109892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9788
W20260927 15:07:51.126262 9716 connection.cc:535] server connection from 127.0.97.1:34973 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:07:51.126586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9919
I20260927 15:07:51.134727 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:51.127579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.0.97.2:43499)
0927 15:07:51.128094 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:51.128102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:51.128129 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:51.128191 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:51.128198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:51.128455 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:51.128598 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:51.128952 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:51.130000 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:51.131329 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:51.131900 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:51.132009 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:51.133044 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:51.133052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:51.133066 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:51.133738 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:51.133792 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:51.133813 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:51.133918 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:51.134125 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 15:07:51.134333 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":282,"threads_started":1}
I20260927 15:07:51.136152 10041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:51.219038 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9919
I20260927 15:07:51.235155 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10049
I20260927 15:07:51.243352 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:51.236297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.97.3:40469)
0927 15:07:51.236957 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:51.236964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:51.236996 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:07:51.237062 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:51.237068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:51.237287 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:51.237421 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:51.237819 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:51.238845 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:51.240153 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:51.240740 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:51.240843 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:51.241815 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:51.241824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:51.241838 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:51.242438 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:51.242482 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:51.242501 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:51.242596 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:51.242749 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:07:51.242898 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":372,"threads_started":1}
I20260927 15:07:51.244573 10171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:51.328464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10049
I20260927 15:07:51.344961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 9698
I20260927 15:07:51.346302 9759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:51.447839 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 9698
2026-09-27T15:07:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2512 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T15:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:51Z Disabled control of system clock
I20260927 15:07:51.522598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44143
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:07:51.868525 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:51.868862 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:51.868947 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:51.876534 10195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:51.876664 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:51.876703 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:51.876752 10195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:51.876789 10195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:51.887208 10195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:51.888528 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:51.890363 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:51.903498 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:51.903590 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:51.903538 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:51.909061 10195 server_base.cc:1034] running on GCE node
I20260927 15:07:51.910096 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:51.911779 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:51.913012 10195 hybrid_clock.cc:648] HybridClock initialized: now 1790521671912955 us; error 49 us; skew 500 ppm
I20260927 15:07:51.913363 10195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:51.916041 10195 webserver.cc:460] Webserver started at http://127.0.97.62:41605/ using document root <none> and password file <none>
I20260927 15:07:51.916765 10195 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:51.916857 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:51.917150 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:51.920244 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "30cc552b790d4d19a8a0214fa75d5439"
format_stamp: "Formatted at 2026-09-27 15:07:51 on dist-test-slave-g0w1"
server_key: "c3682c5cbc4fc270a97229681a8d0387"
server_key_iv: "60aec24eca33db170fa3e0c0a9696269"
server_key_version: "encryptionkey@0"
I20260927 15:07:51.921042 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "30cc552b790d4d19a8a0214fa75d5439"
format_stamp: "Formatted at 2026-09-27 15:07:51 on dist-test-slave-g0w1"
server_key: "c3682c5cbc4fc270a97229681a8d0387"
server_key_iv: "60aec24eca33db170fa3e0c0a9696269"
server_key_version: "encryptionkey@0"
I20260927 15:07:51.926645 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:07:51.930459 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:51.932052 10195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:51.932214 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "30cc552b790d4d19a8a0214fa75d5439"
format_stamp: "Formatted at 2026-09-27 15:07:51 on dist-test-slave-g0w1"
server_key: "c3682c5cbc4fc270a97229681a8d0387"
server_key_iv: "60aec24eca33db170fa3e0c0a9696269"
server_key_version: "encryptionkey@0"
I20260927 15:07:51.932408 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:51.938526 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:51.939479 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:51.939755 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:51.968992 10195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44143
I20260927 15:07:51.969013 10260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44143 every 8 connection(s)
I20260927 15:07:51.970619 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:51.974776 10261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:51.976449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10195
I20260927 15:07:51.976650 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:51.977500 388 external_mini_cluster.cc:1428] Setting key e94206769665e85a8358034230a729ad
I20260927 15:07:51.986832 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:51.979930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.0.97.62:44143)
0927 15:07:51.981129 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:51.981141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:51.981187 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:07:51.982860 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:51.982886 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:51.983430 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:51.983892 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:51.985072 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:51.985078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:51.985091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:51.985105 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:51.985172 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:51.985190 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:51.985478 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:51.985627 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 15:07:51.985889 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":434,"threads_started":1}
I20260927 15:07:51.987540 10261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439: Bootstrap starting.
I20260927 15:07:51.990985 10261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:51.992168 10261 log.cc:824] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:51.995287 10261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439: No bootstrap required, opened a new log
I20260927 15:07:52.002883 10261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cc552b790d4d19a8a0214fa75d5439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260927 15:07:52.003387 10261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:52.003490 10261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cc552b790d4d19a8a0214fa75d5439, State: Initialized, Role: FOLLOWER
I20260927 15:07:52.004104 10261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cc552b790d4d19a8a0214fa75d5439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260927 15:07:52.004341 10261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:52.004477 10261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:52.004660 10261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:52.007221 10261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cc552b790d4d19a8a0214fa75d5439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260927 15:07:52.007784 10261 leader_election.cc:302] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30cc552b790d4d19a8a0214fa75d5439; no voters:
I20260927 15:07:52.008390 10261 leader_election.cc:288] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:52.008577 10266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:52.009384 10266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [term 1 LEADER]: Becoming Leader. State: Replica: 30cc552b790d4d19a8a0214fa75d5439, State: Running, Role: LEADER
I20260927 15:07:52.009987 10266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cc552b790d4d19a8a0214fa75d5439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260927 15:07:52.010746 10261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:52.013177 10268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30cc552b790d4d19a8a0214fa75d5439. Latest consensus state: current_term: 1 leader_uuid: "30cc552b790d4d19a8a0214fa75d5439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cc552b790d4d19a8a0214fa75d5439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } } }
I20260927 15:07:52.013625 10268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:52.013479 10267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30cc552b790d4d19a8a0214fa75d5439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cc552b790d4d19a8a0214fa75d5439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } } }
I20260927 15:07:52.013806 10267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:52.015220 10275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:52.020903 10275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:52.030272 10275 catalog_manager.cc:1347] Generated new cluster ID: f93c61c6b515438ba7bccddb24ac31c9
I20260927 15:07:52.030404 10275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:52.051460 10275 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:52.052034 10275 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:52.060803 10275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30cc552b790d4d19a8a0214fa75d5439: Generated new TSK 0
I20260927 15:07:52.061944 10275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:52.070549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:07:52.419868 10285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:52.420187 10285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:52.420342 10285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:52.427824 10285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:52.427944 10285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:52.428138 10285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:52.438073 10285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:52.439509 10285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:52.441339 10285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:52.455459 10291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.455539 10293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.455509 10290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.456465 10285 server_base.cc:1034] running on GCE node
I20260927 15:07:52.457134 10285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:52.458220 10285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:52.459441 10285 hybrid_clock.cc:648] HybridClock initialized: now 1790521672459373 us; error 66 us; skew 500 ppm
I20260927 15:07:52.459774 10285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:52.462080 10285 webserver.cc:460] Webserver started at http://127.0.97.1:41765/ using document root <none> and password file <none>
I20260927 15:07:52.462852 10285 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:52.462968 10285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:52.463318 10285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:52.466240 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "47f29ba6ca164dc8abd717c88ca75891"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-g0w1"
server_key: "acf5a3b9372cab1e9e7e4100b35cce77"
server_key_iv: "7f6a10f2422e52144f276983e7fe022f"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.466977 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "47f29ba6ca164dc8abd717c88ca75891"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-g0w1"
server_key: "acf5a3b9372cab1e9e7e4100b35cce77"
server_key_iv: "7f6a10f2422e52144f276983e7fe022f"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.472301 10285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:07:52.475855 10299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.477257 10285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:52.477439 10285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "47f29ba6ca164dc8abd717c88ca75891"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-g0w1"
server_key: "acf5a3b9372cab1e9e7e4100b35cce77"
server_key_iv: "7f6a10f2422e52144f276983e7fe022f"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.477697 10285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:52.483539 10285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:52.484373 10285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:52.484649 10285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:52.485558 10285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:52.486941 10285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:52.487108 10285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.487246 10285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:52.487330 10285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.521709 10285 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34877
I20260927 15:07:52.521731 10411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34877 every 8 connection(s)
I20260927 15:07:52.523216 10285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:52.524406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10285
I20260927 15:07:52.524590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:52.525383 388 external_mini_cluster.cc:1428] Setting key 86df89931d068134b4546b2a9976e45d
I20260927 15:07:52.528699 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:07:52.533795 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.526240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41427 (local address 127.0.97.62:44143)
0927 15:07:52.526689 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.526694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.528573 (+ 1879us) server_negotiation.cc:407] Connection header received
0927 15:07:52.531413 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.531418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.531471 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.531557 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.533128 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.533133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.533136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:52.533152 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.533193 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:52.533205 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.533307 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.533529 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.533578 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260927 15:07:52.535385 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.526957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44143 (local address 127.0.97.1:41427)
0927 15:07:52.528384 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.528435 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.531088 (+ 2653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.531856 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.531870 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.532016 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.532733 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:52.532738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.532794 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:52.532916 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.533306 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.533319 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.533330 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.533612 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.533897 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":496,"threads_started":1}
I20260927 15:07:52.537122 10412 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44143
I20260927 15:07:52.537497 10412 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:52.538138 10412 heartbeater.cc:511] Master 127.0.97.62:44143 requested a full tablet report, sending...
I20260927 15:07:52.540490 10226 ts_manager.cc:194] Registered new tserver with Master: 47f29ba6ca164dc8abd717c88ca75891 (127.0.97.1:34877)
W20260927 15:07:52.872512 10416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:52.872835 10416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:52.873023 10416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:52.880575 10416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:52.880697 10416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:52.880924 10416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:52.890704 10416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:52.892103 10416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:52.893991 10416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:52.908691 10424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.908792 10422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.908691 10421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.909502 10416 server_base.cc:1034] running on GCE node
I20260927 15:07:52.910259 10416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:52.911481 10416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:52.912724 10416 hybrid_clock.cc:648] HybridClock initialized: now 1790521672912682 us; error 50 us; skew 500 ppm
I20260927 15:07:52.913072 10416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:52.915781 10416 webserver.cc:460] Webserver started at http://127.0.97.2:46165/ using document root <none> and password file <none>
I20260927 15:07:52.916510 10416 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:52.916640 10416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:52.916975 10416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:52.920135 10416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "b2c65bbfb3484ae7a65af6632e7462a2"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-g0w1"
server_key: "cadacf728693cd6c2c1abe755ec4a996"
server_key_iv: "4e59a12082fbb47f19ed61076760efed"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.920992 10416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b2c65bbfb3484ae7a65af6632e7462a2"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-g0w1"
server_key: "cadacf728693cd6c2c1abe755ec4a996"
server_key_iv: "4e59a12082fbb47f19ed61076760efed"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.926548 10416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 15:07:52.930341 10430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.931900 10416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:52.932109 10416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "b2c65bbfb3484ae7a65af6632e7462a2"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-g0w1"
server_key: "cadacf728693cd6c2c1abe755ec4a996"
server_key_iv: "4e59a12082fbb47f19ed61076760efed"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.932363 10416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:52.938676 10416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:52.939659 10416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:52.939944 10416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:52.940917 10416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:52.942384 10416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:52.942509 10416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.942641 10416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:52.942726 10416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.977494 10416 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39479
I20260927 15:07:52.977557 10542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39479 every 8 connection(s)
I20260927 15:07:52.978922 10416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:52.984879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10416
I20260927 15:07:52.985067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:52.985548 388 external_mini_cluster.cc:1428] Setting key e0f0e558acb9e7460630945f74ee83bc
I20260927 15:07:52.988824 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.982125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57043 (local address 127.0.97.62:44143)
0927 15:07:52.982321 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.982326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.984183 (+ 1857us) server_negotiation.cc:407] Connection header received
0927 15:07:52.986116 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.986122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.986217 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.986332 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.988054 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.988061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.988066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:52.988086 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.988126 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:52.988142 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.988270 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.988451 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.988517 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:52.988845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:07:52.989786 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.982848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44143 (local address 127.0.97.2:57043)
0927 15:07:52.983990 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.984033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.985811 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.986648 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.986660 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.986800 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.987695 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:52.987699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.987757 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:52.987863 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.988249 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.988256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.988262 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.988482 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.988710 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":377,"threads_started":1}
I20260927 15:07:52.991346 10543 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44143
I20260927 15:07:52.991662 10543 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:52.992267 10543 heartbeater.cc:511] Master 127.0.97.62:44143 requested a full tablet report, sending...
I20260927 15:07:52.993731 10226 ts_manager.cc:194] Registered new tserver with Master: b2c65bbfb3484ae7a65af6632e7462a2 (127.0.97.2:39479)
W20260927 15:07:53.341035 10546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:53.341360 10546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:53.341526 10546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:53.349042 10546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:53.349176 10546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:53.349292 10546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:53.358796 10546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:53.360183 10546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:53.362013 10546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:53.375762 10551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:53.375811 10552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:53.375825 10554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:53.376621 10546 server_base.cc:1034] running on GCE node
I20260927 15:07:53.377336 10546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:53.378358 10546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:53.379621 10546 hybrid_clock.cc:648] HybridClock initialized: now 1790521673379564 us; error 50 us; skew 500 ppm
I20260927 15:07:53.379978 10546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:07:53.382535 10546 webserver.cc:460] Webserver started at http://127.0.97.3:37069/ using document root <none> and password file <none>
I20260927 15:07:53.383344 10546 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:53.383474 10546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:53.383792 10546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:53.386782 10546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "7cb68664745548c7961015c652de564f"
format_stamp: "Formatted at 2026-09-27 15:07:53 on dist-test-slave-g0w1"
server_key: "28a9f318e19e516d98092e78ffbf7779"
server_key_iv: "2abf881fd02bcf415caf3a102f1b3490"
server_key_version: "encryptionkey@0"
I20260927 15:07:53.387760 10546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7cb68664745548c7961015c652de564f"
format_stamp: "Formatted at 2026-09-27 15:07:53 on dist-test-slave-g0w1"
server_key: "28a9f318e19e516d98092e78ffbf7779"
server_key_iv: "2abf881fd02bcf415caf3a102f1b3490"
server_key_version: "encryptionkey@0"
I20260927 15:07:53.393499 10546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:53.397215 10560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:53.398806 10546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:53.399022 10546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "7cb68664745548c7961015c652de564f"
format_stamp: "Formatted at 2026-09-27 15:07:53 on dist-test-slave-g0w1"
server_key: "28a9f318e19e516d98092e78ffbf7779"
server_key_iv: "2abf881fd02bcf415caf3a102f1b3490"
server_key_version: "encryptionkey@0"
I20260927 15:07:53.399376 10546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:53.405463 10546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:53.406383 10546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:53.406680 10546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:53.407779 10546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:53.409260 10546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:53.409408 10546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:53.409535 10546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:53.409621 10546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:53.444756 10546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45897
I20260927 15:07:53.444859 10672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45897 every 8 connection(s)
I20260927 15:07:53.446177 10546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:53.454003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10546
I20260927 15:07:53.454221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:53.454794 388 external_mini_cluster.cc:1428] Setting key 0283d932cbb47b47b2230452d5955d53
I20260927 15:07:53.454941 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.449005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33285 (local address 127.0.97.62:44143)
0927 15:07:53.449200 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.449205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.450680 (+ 1475us) server_negotiation.cc:407] Connection header received
0927 15:07:53.452459 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.452465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.452523 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.452629 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.454130 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.454138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.454143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:53.454167 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.454213 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:53.454232 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.454366 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.454564 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.454643 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:07:53.456038 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.449528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44143 (local address 127.0.97.3:33285)
0927 15:07:53.450483 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.450557 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.452162 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.452861 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.452870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.452973 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:53.453805 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:53.453807 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:53.453848 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:53.453938 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:53.454346 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:53.454356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:53.454368 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.454612 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.454827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":281,"threads_started":1}
I20260927 15:07:53.457620 10673 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44143
I20260927 15:07:53.457938 10673 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:53.458582 10673 heartbeater.cc:511] Master 127.0.97.62:44143 requested a full tablet report, sending...
I20260927 15:07:53.460078 10226 ts_manager.cc:194] Registered new tserver with Master: 7cb68664745548c7961015c652de564f (127.0.97.3:45897)
I20260927 15:07:53.470669 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:07:53.476693 10414 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:53.475243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.0.97.62:44143)
0927 15:07:53.475415 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.475420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.475918 (+ 498us) server_negotiation.cc:407] Connection header received
0927 15:07:53.476058 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.476062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.476120 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.476206 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.476481 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37606: BlockingRecv error: recv got EOF from 127.0.0.1:37606 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:53.477718 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10285
I20260927 15:07:53.481335 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.478641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.0.97.1:34877)
0927 15:07:53.479147 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.479162 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.479189 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:53.479271 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.479280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.479523 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.479661 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.480671 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.480678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.480691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:53.480704 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.480754 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:53.480769 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.480878 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.480962 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.481042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":215,"threads_started":1}
I20260927 15:07:53.482556 10407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:53.564584 10412 heartbeater.cc:503] Master 127.0.97.62:44143 was elected leader, sending a full tablet report...
I20260927 15:07:53.565785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10285
W20260927 15:07:53.582401 10211 connection.cc:535] server connection from 127.0.97.1:41427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:07:53.582589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10416
I20260927 15:07:53.586481 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.583798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.0.97.2:39479)
0927 15:07:53.584293 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.584308 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.584335 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:53.584395 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.584401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.584659 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.584789 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.585740 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.585747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.585759 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:53.585776 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.585815 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:53.585829 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.585928 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.586075 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.586197 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260927 15:07:53.587827 10538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:53.669534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10416
I20260927 15:07:53.685979 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10546
I20260927 15:07:53.690124 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.687027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.3:45897)
0927 15:07:53.687736 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.687743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.687774 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:07:53.687836 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.687843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.688116 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.688251 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.689324 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.689331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.689343 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:53.689379 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.689418 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:53.689432 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.689574 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.689701 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.689810 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260927 15:07:53.691437 10668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:53.773813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10546
I20260927 15:07:53.789794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10195
I20260927 15:07:53.791013 10256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:53.892398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10195
2026-09-27T15:07:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2444 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:54 dist-test-slave-g0w1 krb5kdc[10691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:54 dist-test-slave-g0w1 krb5kdc[10691](info): set up 2 sockets
Sep 27 15:07:54 dist-test-slave-g0w1 krb5kdc[10691](info): commencing operation
krb5kdc: starting...
W20260927 15:07:55.024413 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:55 dist-test-slave-g0w1 krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:07:55.263875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:55.613807 10704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:55.614130 10704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:55.614300 10704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:55.621938 10704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:55.622074 10704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:55.622153 10704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:55.622251 10704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:55.622337 10704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:55.633886 10704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:55.635414 10704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:55.636819 10704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:55.649947 10712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:55.649922 10710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:55.650112 10704 server_base.cc:1034] running on GCE node
W20260927 15:07:55.649977 10709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:55.651441 10704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:55.653136 10704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:55.654448 10704 hybrid_clock.cc:648] HybridClock initialized: now 1790521675654362 us; error 43 us; skew 500 ppm
I20260927 15:07:55.654866 10704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:55 dist-test-slave-g0w1 krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:55.658329 10704 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:07:55.660084 10704 webserver.cc:460] Webserver started at http://127.0.97.62:37895/ using document root <none> and password file <none>
I20260927 15:07:55.660802 10704 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:55.660913 10704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:55.661180 10704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:55.664386 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "f8afa27490f7443d87de04867d9c4f80"
format_stamp: "Formatted at 2026-09-27 15:07:55 on dist-test-slave-g0w1"
server_key: "a20b4aa6065b602403bbd6faa48b2959"
server_key_iv: "84ec0212c985f2a09de94685924884d2"
server_key_version: "encryptionkey@0"
I20260927 15:07:55.665154 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8afa27490f7443d87de04867d9c4f80"
format_stamp: "Formatted at 2026-09-27 15:07:55 on dist-test-slave-g0w1"
server_key: "a20b4aa6065b602403bbd6faa48b2959"
server_key_iv: "84ec0212c985f2a09de94685924884d2"
server_key_version: "encryptionkey@0"
I20260927 15:07:55.671133 10704 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:07:55.675225 10719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:55.676820 10704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:55.677037 10704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "f8afa27490f7443d87de04867d9c4f80"
format_stamp: "Formatted at 2026-09-27 15:07:55 on dist-test-slave-g0w1"
server_key: "a20b4aa6065b602403bbd6faa48b2959"
server_key_iv: "84ec0212c985f2a09de94685924884d2"
server_key_version: "encryptionkey@0"
I20260927 15:07:55.677244 10704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:55.689419 10704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:55.694098 10704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:55.694407 10704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:55.713943 10704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39877
I20260927 15:07:55.713984 10770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39877 every 8 connection(s)
I20260927 15:07:55.715453 10704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:55.718128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10704
I20260927 15:07:55.718346 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:07:55.719205 388 external_mini_cluster.cc:1428] Setting key 8821608c2c714a0e2991fcd08ea10373
I20260927 15:07:55.720425 10771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:55.733457 10771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80: Bootstrap starting.
Sep 27 15:07:55 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:55.737563 10771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:55.738783 10771 log.cc:824] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:55.742496 10771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80: No bootstrap required, opened a new log
I20260927 15:07:55.743626 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:55.721454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.0.97.62:39877)
0927 15:07:55.722259 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:55.722272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:55.722313 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:07:55.724001 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:55.724028 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:55.724504 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:55.724958 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:55.726482 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:55.727911 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:55.729258 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:55.730034 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:55.730281 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:55.734897 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:55.734906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:55.734923 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:55.734965 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:55.738609 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:55.739671 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:55.739681 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:55.739691 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:55.739938 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:55.740528 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:55.740532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:55.740534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:55.741098 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:55.741386 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:55.741927 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 15:07:55.742387 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":372,"threads_started":1}
I20260927 15:07:55.750435 10771 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8afa27490f7443d87de04867d9c4f80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39877 } }
I20260927 15:07:55.750957 10771 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:55.751114 10771 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8afa27490f7443d87de04867d9c4f80, State: Initialized, Role: FOLLOWER
I20260927 15:07:55.751897 10771 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8afa27490f7443d87de04867d9c4f80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39877 } }
I20260927 15:07:55.752128 10771 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:55.752223 10771 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:55.752355 10771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:55.754542 10771 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8afa27490f7443d87de04867d9c4f80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39877 } }
I20260927 15:07:55.755029 10771 leader_election.cc:302] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8afa27490f7443d87de04867d9c4f80; no voters:
I20260927 15:07:55.755769 10771 leader_election.cc:288] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:55.756006 10776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:55.757081 10776 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [term 1 LEADER]: Becoming Leader. State: Replica: f8afa27490f7443d87de04867d9c4f80, State: Running, Role: LEADER
I20260927 15:07:55.757702 10776 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8afa27490f7443d87de04867d9c4f80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39877 } }
I20260927 15:07:55.758155 10771 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:55.760565 10778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8afa27490f7443d87de04867d9c4f80. Latest consensus state: current_term: 1 leader_uuid: "f8afa27490f7443d87de04867d9c4f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8afa27490f7443d87de04867d9c4f80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39877 } } }
I20260927 15:07:55.760973 10778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:55.760721 10777 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8afa27490f7443d87de04867d9c4f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8afa27490f7443d87de04867d9c4f80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39877 } } }
I20260927 15:07:55.761142 10777 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:55.763193 10784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:55.768330 10784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:55.777196 10784 catalog_manager.cc:1347] Generated new cluster ID: fb16f42afeee48489b5f4b2ddebd1ce7
I20260927 15:07:55.777336 10784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:55.791115 10784 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:55.792507 10784 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:55.802506 10784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80: Generated new TSK 0
I20260927 15:07:55.803578 10784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:07:55.870525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39877
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:56.222523 10797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:56.222828 10797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:56.222923 10797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:56.230367 10797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:56.230496 10797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:56.230618 10797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:07:56.241283 10797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:56.242664 10797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:56.244371 10797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:56.258563 10803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.258644 10802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.258605 10805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:56.259531 10797 server_base.cc:1034] running on GCE node
I20260927 15:07:56.260202 10797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:56.261245 10797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:56.262513 10797 hybrid_clock.cc:648] HybridClock initialized: now 1790521676262413 us; error 86 us; skew 500 ppm
I20260927 15:07:56.262880 10797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:56 dist-test-slave-g0w1 krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:56.266355 10797 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:07:56.268018 10797 webserver.cc:460] Webserver started at http://127.0.97.1:40087/ using document root <none> and password file <none>
I20260927 15:07:56.268777 10797 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:56.268929 10797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:56.269275 10797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:56.272439 10797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "01d2c705fd0447e999dc4bdaa715d7fb"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-g0w1"
server_key: "150039960bfd9f5d42e778ecd33f27cc"
server_key_iv: "6593af5fa6952374f86138e3ebd06418"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.273248 10797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "01d2c705fd0447e999dc4bdaa715d7fb"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-g0w1"
server_key: "150039960bfd9f5d42e778ecd33f27cc"
server_key_iv: "6593af5fa6952374f86138e3ebd06418"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.278926 10797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:07:56.282912 10812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.284595 10797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:56.284803 10797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "01d2c705fd0447e999dc4bdaa715d7fb"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-g0w1"
server_key: "150039960bfd9f5d42e778ecd33f27cc"
server_key_iv: "6593af5fa6952374f86138e3ebd06418"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.285035 10797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:56.300093 10797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:56.304711 10797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:56.305022 10797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:56.306020 10797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:56.307550 10797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:56.307659 10797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.307813 10797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:56.307889 10797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.341696 10797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32845
I20260927 15:07:56.341743 10924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32845 every 8 connection(s)
I20260927 15:07:56.343022 10797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:56.345361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10797
I20260927 15:07:56.345584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:56.346410 388 external_mini_cluster.cc:1428] Setting key 3f2a13bc21d7b57768cd52c6f9150de6
Sep 27 15:07:56 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:56.365873 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.345766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51381 (local address 127.0.97.62:39877)
0927 15:07:56.346264 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:56.346271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:56.347434 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 15:07:56.349665 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:56.349670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:56.349753 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:56.349879 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:56.351510 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.352226 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.353832 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.354231 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.354372 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.358308 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:56.358317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:56.358324 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:56.358372 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:56.362099 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.363322 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.363327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.363330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.363462 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.364005 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.364009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.364011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.364271 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:56.364380 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:56.365397 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0927 15:07:56.365618 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260927 15:07:56.367511 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.346333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39877 (local address 127.0.97.1:51381)
0927 15:07:56.347262 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:56.347314 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:56.349363 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:56.350138 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:56.350147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:56.350677 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:56.351338 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:56.351359 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.352395 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.352403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:56.353662 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:56.353675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.354467 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.354474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:56.354721 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.355494 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:56.355517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:56.358024 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:56.362353 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:56.362366 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:56.362389 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:56.363126 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:56.363589 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:56.363603 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:56.363610 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:56.363890 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:56.364409 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:56.364421 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:56.364939 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 15:07:56.365272 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 15:07:56.365908 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":330,"threads_started":1}
I20260927 15:07:56.369273 10925 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39877
I20260927 15:07:56.369655 10925 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:56.370440 10925 heartbeater.cc:511] Master 127.0.97.62:39877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:07:56.372918 10736 ts_manager.cc:194] Registered new tserver with Master: 01d2c705fd0447e999dc4bdaa715d7fb (127.0.97.1:32845)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:07:56.374774 10736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:07:56.404070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39877
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:56.747252 10931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:56.747589 10931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:56.747771 10931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:56.755281 10931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:56.755445 10931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:56.755633 10931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:07:56.766119 10931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:56.767565 10931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:56.768889 10931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:56.781996 10936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.782586 10939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.782649 10937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:56.782852 10931 server_base.cc:1034] running on GCE node
I20260927 15:07:56.783586 10931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:56.784586 10931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:56.785837 10931 hybrid_clock.cc:648] HybridClock initialized: now 1790521676785791 us; error 49 us; skew 500 ppm
I20260927 15:07:56.786228 10931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:56 dist-test-slave-g0w1 krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:56.789558 10931 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:07:56.791121 10931 webserver.cc:460] Webserver started at http://127.0.97.2:46137/ using document root <none> and password file <none>
I20260927 15:07:56.791859 10931 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:56.792003 10931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:56.792316 10931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:56.795491 10931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "6b1b7b58dac64677ab6166aefa974859"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-g0w1"
server_key: "3107a2c7c1e66345bef437452d0954fe"
server_key_iv: "8aa7a071af1a301ef4d7a92a6f81d97d"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.796525 10931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6b1b7b58dac64677ab6166aefa974859"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-g0w1"
server_key: "3107a2c7c1e66345bef437452d0954fe"
server_key_iv: "8aa7a071af1a301ef4d7a92a6f81d97d"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.802040 10931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:07:56.805711 10946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.807384 10931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:56.807569 10931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "6b1b7b58dac64677ab6166aefa974859"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-g0w1"
server_key: "3107a2c7c1e66345bef437452d0954fe"
server_key_iv: "8aa7a071af1a301ef4d7a92a6f81d97d"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.807767 10931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:56.842047 10931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:56.846778 10931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:56.847155 10931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:56.848165 10931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:56.849666 10931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:56.849742 10931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.849897 10931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:56.849947 10931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.883747 10931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43789
I20260927 15:07:56.883801 11058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43789 every 8 connection(s)
I20260927 15:07:56.885056 10931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:56.888025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 10931
I20260927 15:07:56.888218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:56.888857 388 external_mini_cluster.cc:1428] Setting key 1b2d88edebcc496f94de1d6f07237ed4
Sep 27 15:07:56 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:56.908624 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.887765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36671 (local address 127.0.97.62:39877)
0927 15:07:56.888227 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:56.888233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:56.889588 (+ 1355us) server_negotiation.cc:407] Connection header received
0927 15:07:56.891954 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:56.891959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:56.892024 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:56.892122 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:56.894360 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.895154 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.897715 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.898138 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.898236 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.902418 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:56.902424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:56.902428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:56.902464 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:56.905245 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.906104 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.906110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.906113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.906275 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.906743 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.906751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.906755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.907028 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:56.907245 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:56.907994 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 15:07:56.908242 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":209,"threads_started":1}
I20260927 15:07:56.909850 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.888442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39877 (local address 127.0.97.2:36671)
0927 15:07:56.889366 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:56.889437 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:56.891638 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:56.892421 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:56.892436 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:56.893278 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:56.894150 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:56.894177 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.895306 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.895314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:56.897541 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:56.897561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.898383 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.898392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:56.898827 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.899833 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:56.899862 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:56.902140 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:56.905444 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:56.905454 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:56.905473 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:56.905953 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:56.906402 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:56.906407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:56.906411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:56.906622 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:56.907217 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:56.907226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:56.907611 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 15:07:56.907912 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 15:07:56.908512 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":377,"threads_started":1}
I20260927 15:07:56.911509 11059 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39877
I20260927 15:07:56.911890 11059 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:56.912878 11059 heartbeater.cc:511] Master 127.0.97.62:39877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:07:56.914669 10736 ts_manager.cc:194] Registered new tserver with Master: 6b1b7b58dac64677ab6166aefa974859 (127.0.97.2:43789)
I20260927 15:07:56.915755 10736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:07:56.942963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39877
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:57.285768 11065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:57.286176 11065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:57.286362 11065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:57.294144 11065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:57.294268 11065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:57.294474 11065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:07:57.304589 11065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:07:57.306007 11065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:57.307498 11065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:57.322503 11070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.322587 11073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.322468 11071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.323572 11065 server_base.cc:1034] running on GCE node
I20260927 15:07:57.324323 11065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:57.325416 11065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:57.326709 11065 hybrid_clock.cc:648] HybridClock initialized: now 1790521677326660 us; error 54 us; skew 500 ppm
I20260927 15:07:57.327354 11065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:07:57 dist-test-slave-g0w1 krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:57.331302 11065 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:07:57.333051 11065 webserver.cc:460] Webserver started at http://127.0.97.3:36541/ using document root <none> and password file <none>
I20260927 15:07:57.333841 11065 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:57.333997 11065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:57.334327 11065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:57.337654 11065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "e332939ec75b4f129795789ce602b991"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-g0w1"
server_key: "bd9e097d06ac32f09b374c8e8380b6a2"
server_key_iv: "0959653fb83f8f71f7c7cd0e472f35f1"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.338583 11065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e332939ec75b4f129795789ce602b991"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-g0w1"
server_key: "bd9e097d06ac32f09b374c8e8380b6a2"
server_key_iv: "0959653fb83f8f71f7c7cd0e472f35f1"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.345423 11065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260927 15:07:57.349458 11080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.351298 11065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:57.351505 11065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "e332939ec75b4f129795789ce602b991"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-g0w1"
server_key: "bd9e097d06ac32f09b374c8e8380b6a2"
server_key_iv: "0959653fb83f8f71f7c7cd0e472f35f1"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.351694 11065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:57.365873 11065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:57.370632 11065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:57.370965 11065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:57.372128 11065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:57.373592 11065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:57.373668 11065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.373797 11065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:57.373849 11065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.378209 10925 heartbeater.cc:503] Master 127.0.97.62:39877 was elected leader, sending a full tablet report...
I20260927 15:07:57.409940 11065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39929
I20260927 15:07:57.409965 11192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39929 every 8 connection(s)
I20260927 15:07:57.411366 11065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:57.416373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 11065
I20260927 15:07:57.416567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:57.417106 388 external_mini_cluster.cc:1428] Setting key 97b423572c8618dab11d66a4a9aa9c88
Sep 27 15:07:57 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:07:57.433797 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.414596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48877 (local address 127.0.97.62:39877)
0927 15:07:57.415180 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.415193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.416360 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 15:07:57.418726 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.418731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.418842 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.418966 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.420597 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.421370 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.423631 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.424048 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.424199 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.427958 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.427964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.427967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.427999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:57.430518 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.431441 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.431446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.431448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.431573 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.432024 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.432033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.432035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.432297 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:57.432459 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.433193 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.433422 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260927 15:07:57.434791 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.415340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39877 (local address 127.0.97.3:48877)
0927 15:07:57.416173 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.416220 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.418358 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.419266 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.419276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.419818 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:57.420422 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.420439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.421541 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.421546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.423474 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.423487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.424295 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.424302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.424553 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.425306 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:57.425329 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:57.427704 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:57.430741 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:57.430755 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:57.430771 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:57.431296 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:57.431692 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:57.431697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:57.431701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:57.431901 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:57.432435 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:57.432443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:57.432814 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.433082 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.433660 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":349,"threads_started":1}
I20260927 15:07:57.436269 11193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39877
I20260927 15:07:57.436642 11193 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:57.437402 11193 heartbeater.cc:511] Master 127.0.97.62:39877 requested a full tablet report, sending...
I20260927 15:07:57.438865 10736 ts_manager.cc:194] Registered new tserver with Master: e332939ec75b4f129795789ce602b991 (127.0.97.3:39929)
I20260927 15:07:57.439744 10736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48877
I20260927 15:07:57.445712 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:07:57.463806 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.450487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.0.97.62:39877)
0927 15:07:57.450644 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.450649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.451242 (+ 593us) server_negotiation.cc:407] Connection header received
0927 15:07:57.451384 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.451387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.451463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.451548 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.452917 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.453638 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.454910 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.455823 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.455945 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.458435 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.458441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.458444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.458479 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:57.461213 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.461849 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.461854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.461857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.461976 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.462379 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.462382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.462384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.462768 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:57.462969 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.463283 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.463480 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:07:57.468071 10736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:57.475152 10736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:57.501893 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.488217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.0.97.1:32845)
0927 15:07:57.493627 (+ 5410us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.493637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.493667 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:07:57.493759 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.493766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.493996 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.494203 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.494948 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.496543 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.499215 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.500644 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.500817 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.501065 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.501350 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.501520 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5232,"thread_start_us":5083,"threads_started":1}
I20260927 15:07:57.502326 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.488141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32845 (local address 127.0.0.1:40492)
0927 15:07:57.489523 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.489576 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.489785 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.494188 (+ 4403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.494192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.494223 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.494694 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.494737 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.496726 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.496732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.499009 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.499020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.500866 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.500872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.501095 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.501153 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.501304 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.501520 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":5,"thread_start_us":943,"threads_started":1}
I20260927 15:07:57.502523 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.489104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.0.97.3:39929)
0927 15:07:57.490903 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.490915 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.492998 (+ 2083us) server_negotiation.cc:407] Connection header received
0927 15:07:57.493235 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.493245 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.493540 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.493761 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.494870 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.496873 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.499457 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.500944 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.501050 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.501267 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.502025 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.502182 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1592,"thread_start_us":1432,"threads_started":1}
I20260927 15:07:57.502907 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.488473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39929 (local address 127.0.0.1:50258)
0927 15:07:57.492869 (+ 4396us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.492888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.493036 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.493811 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.493822 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.493876 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.494474 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.494494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.497017 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.497022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.499279 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.499293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.501457 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.501463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.501647 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.501681 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.501757 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.501886 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4291,"mutex_wait_us":27,"thread_start_us":4148,"threads_started":1}
I20260927 15:07:57.504896 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.490128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43789 (local address 127.0.0.1:50404)
0927 15:07:57.496450 (+ 6322us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.496476 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.496620 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.497331 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.497335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.497361 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.498020 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.498033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.499952 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.499957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.502429 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.502447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.504089 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.504097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.504271 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.504301 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.504389 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.504524 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6195,"mutex_wait_us":72,"thread_start_us":6067,"threads_started":1}
I20260927 15:07:57.505615 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.490668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.0.97.2:43789)
0927 15:07:57.494821 (+ 4153us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.494832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.496601 (+ 1769us) server_negotiation.cc:407] Connection header received
0927 15:07:57.496826 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.496832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.497093 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.497320 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.498254 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.499780 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.502689 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.503881 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.504021 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.504209 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.505180 (+ 971us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.505343 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3994,"thread_start_us":3873,"threads_started":1}
I20260927 15:07:57.506785 11128 tablet_service.cc:1467] Processing CreateTablet for tablet 5431d0197e5d40bd8fbbde11e7c54112 (DEFAULT_TABLE table=test-table [id=bb83011bd1b441a5891f4d9f9b3314ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:57.506709 10860 tablet_service.cc:1467] Processing CreateTablet for tablet 5431d0197e5d40bd8fbbde11e7c54112 (DEFAULT_TABLE table=test-table [id=bb83011bd1b441a5891f4d9f9b3314ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:57.508023 11128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5431d0197e5d40bd8fbbde11e7c54112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:57.508162 10860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5431d0197e5d40bd8fbbde11e7c54112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:57.509336 10994 tablet_service.cc:1467] Processing CreateTablet for tablet 5431d0197e5d40bd8fbbde11e7c54112 (DEFAULT_TABLE table=test-table [id=bb83011bd1b441a5891f4d9f9b3314ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:57.511514 10994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5431d0197e5d40bd8fbbde11e7c54112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:57.520632 11212 tablet_bootstrap.cc:492] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Bootstrap starting.
I20260927 15:07:57.522809 11213 tablet_bootstrap.cc:492] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Bootstrap starting.
I20260927 15:07:57.524627 11212 tablet_bootstrap.cc:654] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:57.525962 11212 log.cc:824] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:57.526394 11213 tablet_bootstrap.cc:654] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:57.526433 11214 tablet_bootstrap.cc:492] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Bootstrap starting.
I20260927 15:07:57.527933 11213 log.cc:824] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:57.530320 11214 tablet_bootstrap.cc:654] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:57.531226 11213 tablet_bootstrap.cc:492] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: No bootstrap required, opened a new log
I20260927 15:07:57.531495 11213 ts_tablet_manager.cc:1397] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 15:07:57.531540 11212 tablet_bootstrap.cc:492] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: No bootstrap required, opened a new log
I20260927 15:07:57.531598 11214 log.cc:824] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:57.531814 11212 ts_tablet_manager.cc:1397] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:07:57.534714 11214 tablet_bootstrap.cc:492] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: No bootstrap required, opened a new log
I20260927 15:07:57.534970 11214 ts_tablet_manager.cc:1397] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 15:07:57.538960 11213 raft_consensus.cc:348] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.539474 11213 raft_consensus.cc:374] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:57.539578 11213 raft_consensus.cc:729] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d2c705fd0447e999dc4bdaa715d7fb, State: Initialized, Role: FOLLOWER
I20260927 15:07:57.540272 11213 consensus_queue.cc:260] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.542215 11213 ts_tablet_manager.cc:1428] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:07:57.543700 11212 raft_consensus.cc:348] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.544092 11212 raft_consensus.cc:374] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:57.544214 11212 raft_consensus.cc:729] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e332939ec75b4f129795789ce602b991, State: Initialized, Role: FOLLOWER
I20260927 15:07:57.544845 11212 consensus_queue.cc:260] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.544989 11214 raft_consensus.cc:348] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.545490 11214 raft_consensus.cc:374] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:57.545607 11214 raft_consensus.cc:729] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1b7b58dac64677ab6166aefa974859, State: Initialized, Role: FOLLOWER
I20260927 15:07:57.546316 11214 consensus_queue.cc:260] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.546528 11212 ts_tablet_manager.cc:1428] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20260927 15:07:57.546723 11193 heartbeater.cc:503] Master 127.0.97.62:39877 was elected leader, sending a full tablet report...
I20260927 15:07:57.547479 11059 heartbeater.cc:503] Master 127.0.97.62:39877 was elected leader, sending a full tablet report...
I20260927 15:07:57.548171 11214 ts_tablet_manager.cc:1428] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260927 15:07:57.580646 11218 raft_consensus.cc:482] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:57.580981 11218 raft_consensus.cc:504] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.582995 11218 leader_election.cc:288] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1b7b58dac64677ab6166aefa974859 (127.0.97.2:43789), e332939ec75b4f129795789ce602b991 (127.0.97.3:39929)
I20260927 15:07:57.587760 11219 raft_consensus.cc:482] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:57.588165 11219 raft_consensus.cc:504] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.591245 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.583826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46091 (local address 127.0.97.3:39929)
0927 15:07:57.584041 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.584046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.584283 (+ 237us) server_negotiation.cc:407] Connection header received
0927 15:07:57.584531 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.584536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.584608 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.584734 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.585427 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.586172 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.588950 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.590041 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.590195 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.590319 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.590780 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.590915 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:07:57.591378 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.583307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39929 (local address 127.0.97.1:46091)
0927 15:07:57.584147 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.584174 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.584374 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.584761 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.584770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.584803 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.585270 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.585285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.586294 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.586299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.588778 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.588790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.590302 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.590309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.590570 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.590613 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.590752 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.590904 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":2,"thread_start_us":468,"threads_started":1}
I20260927 15:07:57.592911 11219 leader_election.cc:288] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d2c705fd0447e999dc4bdaa715d7fb (127.0.97.1:32845), 6b1b7b58dac64677ab6166aefa974859 (127.0.97.2:43789)
I20260927 15:07:57.593029 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5431d0197e5d40bd8fbbde11e7c54112" candidate_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e332939ec75b4f129795789ce602b991" is_pre_election: true
I20260927 15:07:57.593564 11148 raft_consensus.cc:2454] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d2c705fd0447e999dc4bdaa715d7fb in term 0.
I20260927 15:07:57.594266 10814 leader_election.cc:302] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d2c705fd0447e999dc4bdaa715d7fb, e332939ec75b4f129795789ce602b991; no voters:
I20260927 15:07:57.594786 11218 raft_consensus.cc:2781] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:57.594954 11218 raft_consensus.cc:482] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:57.595104 11218 raft_consensus.cc:3037] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 0 FOLLOWER]: Advancing to term 1
W20260927 15:07:57.595417 10926 tablet.cc:2369] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:57.597318 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.583406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59949 (local address 127.0.97.2:43789)
0927 15:07:57.583651 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.583662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.585090 (+ 1428us) server_negotiation.cc:407] Connection header received
0927 15:07:57.585203 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.585210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.585295 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.585398 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.586937 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.588215 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.593228 (+ 5013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.594501 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.594639 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.594730 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.596867 (+ 2137us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.597015 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":75}
I20260927 15:07:57.598096 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.583394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43789 (local address 127.0.97.1:59949)
0927 15:07:57.584973 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.584990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.585093 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.586310 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.586314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.586334 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.586733 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.586743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.588391 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.588399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.591062 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.591097 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.596466 (+ 5369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.596473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.596673 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.596707 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.597488 (+ 781us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.597649 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":60,"thread_start_us":1267,"threads_started":1}
I20260927 15:07:57.598870 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.590793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58729 (local address 127.0.97.1:32845)
0927 15:07:57.590973 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.590978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.591802 (+ 824us) server_negotiation.cc:407] Connection header received
0927 15:07:57.591882 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.591890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.591981 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.592087 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.592780 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.593787 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.596263 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.597557 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.597697 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.597806 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.598507 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.598631 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:07:57.599043 11218 raft_consensus.cc:504] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.599202 11014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5431d0197e5d40bd8fbbde11e7c54112" candidate_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1b7b58dac64677ab6166aefa974859" is_pre_election: true
I20260927 15:07:57.599575 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.590683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32845 (local address 127.0.97.3:58729)
0927 15:07:57.591527 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.591550 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.591679 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.592277 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.592285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.592323 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.592591 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.592605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.593960 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.593967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.596048 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.596062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.597965 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.597971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.598189 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.598227 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.599140 (+ 913us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.599299 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720}
I20260927 15:07:57.599824 11014 raft_consensus.cc:2454] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d2c705fd0447e999dc4bdaa715d7fb in term 0.
I20260927 15:07:57.600510 10880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5431d0197e5d40bd8fbbde11e7c54112" candidate_uuid: "e332939ec75b4f129795789ce602b991" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" is_pre_election: true
I20260927 15:07:57.600646 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5431d0197e5d40bd8fbbde11e7c54112" candidate_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e332939ec75b4f129795789ce602b991"
I20260927 15:07:57.600870 11148 raft_consensus.cc:3037] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:57.601029 10880 raft_consensus.cc:2379] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e332939ec75b4f129795789ce602b991 in current term 1: Already voted for candidate 01d2c705fd0447e999dc4bdaa715d7fb in this term.
I20260927 15:07:57.601111 11014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5431d0197e5d40bd8fbbde11e7c54112" candidate_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1b7b58dac64677ab6166aefa974859"
I20260927 15:07:57.601303 11014 raft_consensus.cc:3037] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:57.602219 11218 leader_election.cc:288] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [CANDIDATE]: Term 1 election: Requested vote from peers 6b1b7b58dac64677ab6166aefa974859 (127.0.97.2:43789), e332939ec75b4f129795789ce602b991 (127.0.97.3:39929)
I20260927 15:07:57.603221 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.591501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35325 (local address 127.0.97.2:43789)
0927 15:07:57.592295 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.592303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.593434 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 15:07:57.593602 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.593608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.593788 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.593896 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:57.594972 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.596177 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.599599 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.601734 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.601880 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.602038 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.602679 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.602829 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":650,"thread_start_us":1572,"threads_started":1}
I20260927 15:07:57.603237 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.591318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43789 (local address 127.0.97.3:35325)
0927 15:07:57.593114 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.593139 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.593274 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.594095 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.594100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.594125 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:57.594771 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.594789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.596426 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.596432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.599249 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.599265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.602202 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.602209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.602482 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.602528 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.602635 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.602801 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1557,"thread_start_us":1350,"threads_started":1}
I20260927 15:07:57.604254 11013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5431d0197e5d40bd8fbbde11e7c54112" candidate_uuid: "e332939ec75b4f129795789ce602b991" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1b7b58dac64677ab6166aefa974859" is_pre_election: true
I20260927 15:07:57.604521 11014 raft_consensus.cc:2454] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d2c705fd0447e999dc4bdaa715d7fb in term 1.
I20260927 15:07:57.604674 11148 raft_consensus.cc:2454] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d2c705fd0447e999dc4bdaa715d7fb in term 1.
I20260927 15:07:57.605012 10813 leader_election.cc:302] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d2c705fd0447e999dc4bdaa715d7fb, 6b1b7b58dac64677ab6166aefa974859; no voters:
I20260927 15:07:57.605208 11081 leader_election.cc:302] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e332939ec75b4f129795789ce602b991; no voters: 01d2c705fd0447e999dc4bdaa715d7fb, 6b1b7b58dac64677ab6166aefa974859
I20260927 15:07:57.605357 11218 raft_consensus.cc:2781] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:57.605695 11219 raft_consensus.cc:2726] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 15:07:57.606025 11218 raft_consensus.cc:686] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 1 LEADER]: Becoming Leader. State: Replica: 01d2c705fd0447e999dc4bdaa715d7fb, State: Running, Role: LEADER
I20260927 15:07:57.606556 11218 consensus_queue.cc:237] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } }
I20260927 15:07:57.611939 10736 catalog_manager.cc:5495] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d2c705fd0447e999dc4bdaa715d7fb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d2c705fd0447e999dc4bdaa715d7fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 } health_report { overall_health: UNKNOWN } } }
W20260927 15:07:57.637166 11060 tablet.cc:2369] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:57.655623 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.649389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.97.1:32845)
0927 15:07:57.649603 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.649611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.649793 (+ 182us) server_negotiation.cc:407] Connection header received
0927 15:07:57.650069 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.650073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.650180 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.650351 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:57.650785 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.651709 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.653163 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.653637 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.653783 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.654336 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:57.654898 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:57.655108 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.655166 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.655256 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:07:57.663643 11194 tablet.cc:2369] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:57.664137 11013 raft_consensus.cc:1261] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 1 FOLLOWER]: Refusing update from remote peer 01d2c705fd0447e999dc4bdaa715d7fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:57.664219 11148 raft_consensus.cc:1261] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 1 FOLLOWER]: Refusing update from remote peer 01d2c705fd0447e999dc4bdaa715d7fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:57.665169 11218 consensus_queue.cc:1035] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e332939ec75b4f129795789ce602b991" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:57.665458 11225 consensus_queue.cc:1035] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1b7b58dac64677ab6166aefa974859" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:57.693450 10735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:57.693763 10735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:57.697734 10735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8afa27490f7443d87de04867d9c4f80: Sending DeleteTablet for 3 replicas of tablet 5431d0197e5d40bd8fbbde11e7c54112
I20260927 15:07:57.699254 10860 tablet_service.cc:1514] Processing DeleteTablet for tablet 5431d0197e5d40bd8fbbde11e7c54112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40492
I20260927 15:07:57.699240 10994 tablet_service.cc:1514] Processing DeleteTablet for tablet 5431d0197e5d40bd8fbbde11e7c54112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50404
I20260927 15:07:57.700060 11128 tablet_service.cc:1514] Processing DeleteTablet for tablet 5431d0197e5d40bd8fbbde11e7c54112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50258
I20260927 15:07:57.701382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10797
I20260927 15:07:57.707742 11239 tablet_replica.cc:330] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: stopping tablet replica
I20260927 15:07:57.708419 11239 raft_consensus.cc:2229] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:57.709189 11239 raft_consensus.cc:2258] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:57.710300 11240 tablet_replica.cc:330] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: stopping tablet replica
I20260927 15:07:57.710851 11240 raft_consensus.cc:2229] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:57.712700 11240 raft_consensus.cc:2258] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:57.713372 11239 ts_tablet_manager.cc:1905] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:57.717528 11240 ts_tablet_manager.cc:1905] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:57.721714 11239 ts_tablet_manager.cc:1918] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:57.722045 11239 log.cc:1196] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/5431d0197e5d40bd8fbbde11e7c54112
I20260927 15:07:57.722112 11238 tablet_replica.cc:330] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: stopping tablet replica
I20260927 15:07:57.722596 11238 raft_consensus.cc:2229] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:57.722859 11239 ts_tablet_manager.cc:1939] T 5431d0197e5d40bd8fbbde11e7c54112 P 6b1b7b58dac64677ab6166aefa974859: Deleting consensus metadata
I20260927 15:07:57.723235 11238 raft_consensus.cc:2258] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:57.725075 10720 catalog_manager.cc:4841] TS 6b1b7b58dac64677ab6166aefa974859 (127.0.97.2:43789): tablet 5431d0197e5d40bd8fbbde11e7c54112 (table test-table [id=bb83011bd1b441a5891f4d9f9b3314ec]) successfully deleted
Sep 27 15:07:57 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:07:57.728034 11238 ts_tablet_manager.cc:1905] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:57.731966 11240 ts_tablet_manager.cc:1918] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:57.732271 11240 log.cc:1196] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/5431d0197e5d40bd8fbbde11e7c54112
I20260927 15:07:57.732913 11240 ts_tablet_manager.cc:1939] T 5431d0197e5d40bd8fbbde11e7c54112 P e332939ec75b4f129795789ce602b991: Deleting consensus metadata
I20260927 15:07:57.735121 10721 catalog_manager.cc:4841] TS e332939ec75b4f129795789ce602b991 (127.0.97.3:39929): tablet 5431d0197e5d40bd8fbbde11e7c54112 (table test-table [id=bb83011bd1b441a5891f4d9f9b3314ec]) successfully deleted
I20260927 15:07:57.741693 11238 ts_tablet_manager.cc:1918] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:57.741887 11238 log.cc:1196] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/5431d0197e5d40bd8fbbde11e7c54112
I20260927 15:07:57.742744 11238 ts_tablet_manager.cc:1939] T 5431d0197e5d40bd8fbbde11e7c54112 P 01d2c705fd0447e999dc4bdaa715d7fb: Deleting consensus metadata
I20260927 15:07:57.742980 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.703053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.0.97.1:32845)
0927 15:07:57.703313 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.703322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.714530 (+ 11208us) server_negotiation.cc:407] Connection header received
0927 15:07:57.714971 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.714977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.715113 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.715186 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.717040 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.718141 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.720099 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.720766 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.720881 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.734151 (+ 13270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.734169 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.734188 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.734240 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:57.738440 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.739685 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.739696 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.739700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.739883 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.740433 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.740439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.740442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.741015 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:57.741387 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.742316 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.742561 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:07:57.743774 10920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:57.841787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10797
I20260927 15:07:57.841841 10720 catalog_manager.cc:4841] TS 01d2c705fd0447e999dc4bdaa715d7fb (127.0.97.1:32845): tablet 5431d0197e5d40bd8fbbde11e7c54112 (table test-table [id=bb83011bd1b441a5891f4d9f9b3314ec]) successfully deleted
I20260927 15:07:57.860709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10931
Sep 27 15:07:57 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:07:57.876925 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.861492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.0.97.2:43789)
0927 15:07:57.861659 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.861664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.861685 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:07:57.861863 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.861867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.861965 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.862083 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.863554 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.864275 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.865650 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.866052 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.866165 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.870501 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.870517 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.870535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.870586 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:57.873777 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.874602 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.874610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.874612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.874757 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.875286 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.875290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.875292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.875667 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:57.875972 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.876325 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.876577 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:07:57.877662 11054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:57.971460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10931
I20260927 15:07:57.989012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 11065
Sep 27 15:07:57 dist-test-slave-g0w1 krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:07:58.004391 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.989820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.0.97.3:39929)
0927 15:07:57.990009 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.990013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.990036 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:07:57.990220 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.990224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.990288 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.990373 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.991659 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.992393 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.993693 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.994108 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.994221 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.998656 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.998672 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.998688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.998732 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:58.001704 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:58.002380 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:58.002387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:58.002389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:58.002520 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:58.002903 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:58.002910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:58.002916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:58.003433 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:58.003663 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:58.003963 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 15:07:58.004128 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:07:58.005118 11188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:58.097023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 11065
I20260927 15:07:58.115535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 10704
I20260927 15:07:58.116662 10766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:07:58.228039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 10704
2026-09-27T15:07:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4338 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:58 dist-test-slave-g0w1 krb5kdc[11248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:58 dist-test-slave-g0w1 krb5kdc[11248](info): set up 2 sockets
Sep 27 15:07:58 dist-test-slave-g0w1 krb5kdc[11248](info): commencing operation
krb5kdc: starting...
W20260927 15:07:59.348348 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:07:59 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 15:07:59.686725 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0r1zNn/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/postgres -l logfile start
2026-09-27 15:08:02.376 UTC [11278] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:08:02.377 UTC [11278] LOG: listening on IPv4 address "127.0.97.20", port 50491
2026-09-27 15:08:02.380 UTC [11278] LOG: listening on Unix socket "/tmp/.s.PGSQL.50491"
2026-09-27 15:08:02.404 UTC [11281] LOG: database system was shut down at 2026-09-27 15:08:00 UTC
2026-09-27 15:08:02.408 UTC [11278] LOG: database system is ready to accept connections
I20260927 15:08:03.379789 388 mini_postgres.cc:96] Postgres bound to 50491
2026-09-27 15:08:03.397 UTC [11289] FATAL: database "slave" does not exist
127.0.97.20:50491 - accepting connections
I20260927 15:08:03.397959 388 mini_ranger.cc:161] Starting Ranger...
I20260927 15:08:03.424871 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:08:03.960187 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:08:03.960343 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-admin
2026-09-27 15:08:04,280 [I] DB FLAVOR :POSTGRES
2026-09-27 15:08:04,280 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:08:04,280 [I] Checking connection..
2026-09-27 15:08:04,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:08:04,543 [I] Checking connection passed.
2026-09-27 15:08:04,544 [I] --------- Verifying version history table ---------
2026-09-27 15:08:04,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:04,823 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:08:04,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:05,073 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:08:05,073 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:08:05,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:08:05.315 UTC [11353] WARNING: there is no transaction in progress
2026-09-27 15:08:05,321 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:08:05,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:05,578 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:08:05,578 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:08:05,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:08:05,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:08:06,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g0w1.c.gcp-upstream.internal','N') ;"
2026-09-27 15:08:06,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:08:06,566 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:08:06,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:08:06,817 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:08:06,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:08:07,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:08:07,294 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:08:07,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:08:08.317 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.334 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.353 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.370 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.388 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.407 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.467 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.474 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.490 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.501 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.513 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.527 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.538 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.545 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.553 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:08.560 UTC [11461] WARNING: there is no transaction in progress
2026-09-27 15:08:09,284 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:08:09,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:08:09,535 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:08:09,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:08:09,798 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:08:09,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:08:10,042 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:08:10,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:08:10,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:08:10,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g0w1.c.gcp-upstream.internal';"
2026-09-27 15:08:10,817 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:08:10,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:08:11,061 [I] DB_PATCHES have already been applied
I20260927 15:08:11.068037 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0r1zNn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:08:11.068179 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 15:08:11.084318 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:33197
Sep 27, 2026 3:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:08:12.226424 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.828s user 0.000s sys 0.046s
Sep 27, 2026 3:08:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:08:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:08:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:08:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:08:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:08:35.693631 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:08:35 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521715, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:08:35.866321 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:08:35.888890 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:08:36.891307 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:08:36.891449 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms
I20260927 15:08:36.891598 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:50491
2026-09-27 15:08:37,324 [I] DB FLAVOR :POSTGRES
2026-09-27 15:08:37,324 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:08:37,324 [I] Checking connection
2026-09-27 15:08:37,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:08:37,587 [I] connection success
2026-09-27 15:08:37,587 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:08:37,588 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:08:37,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:08:37,854 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:08:37,855 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:08:37,855 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:08:37,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:08:38,167 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:08:38.735813 388 mini_ranger_kms.cc:310] Created kms service
I20260927 15:08:38.963336 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:08:39.013736 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:08:39 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521715, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:08:39.499898 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:08:39.500293 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:08:39.500365 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 15:08:39.520231 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11642
I20260927 15:08:39.520466 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:34731
Sep 27, 2026 3:08:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:08:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:08:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:08:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:08:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:08:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:08:41.859964 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.994s user 0.090s sys 0.359s
I20260927 15:08:41.860227 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:08:41.860423 388 mini_ranger_kms.cc:401] 127.0.97.20:34731/kms/v1/keys
Sep 27 15:08:41 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521721, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:08:42 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521721, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 15:08:42 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521721, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 15:08:45 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:08:45.399632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43023
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43023
--ranger_config_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34731/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:08:45.769114 11692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:45.769454 11692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:45.769544 11692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:45.777071 11692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:08:45.777194 11692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:45.777248 11692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:45.777297 11692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:08:45.777346 11692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:08:45.788244 11692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:34731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43023
--ranger_config_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:08:45.789621 11692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:45.790961 11692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:45.804111 11697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:45.804154 11698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:45.804385 11700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:45.805766 11692 server_base.cc:1034] running on GCE node
I20260927 15:08:45.806694 11692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:45.808372 11692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:45.809614 11692 hybrid_clock.cc:648] HybridClock initialized: now 1790521725809571 us; error 35 us; skew 500 ppm
I20260927 15:08:45.809983 11692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:08:45 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:45.813496 11692 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:08:45.815228 11692 webserver.cc:460] Webserver started at http://127.0.97.62:35939/ using document root <none> and password file <none>
I20260927 15:08:45.815914 11692 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:45.816013 11692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:45.816265 11692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:08:45 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:08:45.878139 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "f3b2a84a2cff47e4bbf152692b35b748"
format_stamp: "Formatted at 2026-09-27 15:08:45 on dist-test-slave-g0w1"
server_key: "eb6bf130b5d52aa7f1af00892813edab"
server_key_iv: "579dd13f8f8d72d2cc023cd243b4d21c"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:45.879149 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3b2a84a2cff47e4bbf152692b35b748"
format_stamp: "Formatted at 2026-09-27 15:08:45 on dist-test-slave-g0w1"
server_key: "eb6bf130b5d52aa7f1af00892813edab"
server_key_iv: "579dd13f8f8d72d2cc023cd243b4d21c"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:45.885245 11692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:08:45.906324 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:45.908120 11692 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:08:45.908389 11692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "f3b2a84a2cff47e4bbf152692b35b748"
format_stamp: "Formatted at 2026-09-27 15:08:45 on dist-test-slave-g0w1"
server_key: "eb6bf130b5d52aa7f1af00892813edab"
server_key_iv: "579dd13f8f8d72d2cc023cd243b4d21c"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:45.908641 11692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:45.935649 11692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:45.940414 11692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:45.940762 11692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:45.969584 11692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43023
I20260927 15:08:45.969630 11758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43023 every 8 connection(s)
I20260927 15:08:45.971058 11692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:08:45.975723 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:45.979987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 11692
I20260927 15:08:45.980216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:08:45.981022 388 external_mini_cluster.cc:1428] Setting key c141db1a9fff008ddb852aa30239c781
I20260927 15:08:45.989447 11759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748: Bootstrap starting.
I20260927 15:08:45.994221 11759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:45.995716 11759 log.cc:824] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:45.999401 11759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748: No bootstrap required, opened a new log
Sep 27 15:08:45 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:08:46.011027 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:45.983629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.0.97.62:43023)
0927 15:08:45.984578 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:45.984592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:45.984639 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:08:45.987239 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:45.987269 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:45.987776 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:45.988168 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:45.990018 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:45.992415 (+ 2397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:45.994598 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:45.995588 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:45.995758 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:46.000784 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:46.000797 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:46.000818 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:46.000874 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:46.005099 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:46.006027 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:46.006038 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:46.006049 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:46.006277 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:46.007272 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:46.007278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:46.007284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:46.008123 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:46.008436 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:46.008957 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:08:46.009473 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":470,"threads_started":1}
I20260927 15:08:46.011217 11759 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b2a84a2cff47e4bbf152692b35b748" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43023 } }
I20260927 15:08:46.011714 11759 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:46.011842 11759 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b2a84a2cff47e4bbf152692b35b748, State: Initialized, Role: FOLLOWER
I20260927 15:08:46.012555 11759 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b2a84a2cff47e4bbf152692b35b748" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43023 } }
I20260927 15:08:46.012965 11759 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:08:46.013134 11759 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:08:46.013353 11759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:46.016237 11759 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b2a84a2cff47e4bbf152692b35b748" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43023 } }
I20260927 15:08:46.016815 11759 leader_election.cc:302] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b2a84a2cff47e4bbf152692b35b748; no voters:
I20260927 15:08:46.017472 11759 leader_election.cc:288] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:08:46.017607 11764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:08:46.019093 11764 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [term 1 LEADER]: Becoming Leader. State: Replica: f3b2a84a2cff47e4bbf152692b35b748, State: Running, Role: LEADER
I20260927 15:08:46.019675 11764 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b2a84a2cff47e4bbf152692b35b748" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43023 } }
I20260927 15:08:46.020015 11759 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:08:46.023213 11766 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b2a84a2cff47e4bbf152692b35b748. Latest consensus state: current_term: 1 leader_uuid: "f3b2a84a2cff47e4bbf152692b35b748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b2a84a2cff47e4bbf152692b35b748" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43023 } } }
I20260927 15:08:46.023710 11766 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:46.023763 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b2a84a2cff47e4bbf152692b35b748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b2a84a2cff47e4bbf152692b35b748" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43023 } } }
I20260927 15:08:46.024118 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748 [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:46.027673 11772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:08:46.035578 11772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:08:46.036276 11759 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:08:46.055475 11772 catalog_manager.cc:1347] Generated new cluster ID: 9be0248c8e6b4ee1abbb4e17d7fbbe6e
I20260927 15:08:46.055667 11772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:08:46.082279 11772 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:08:46.084441 11772 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:08:46.093729 11772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748: Generated new TSK 0
I20260927 15:08:46.095053 11772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:08:46.839713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43023
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:08:47 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:08:47.336647 11804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:47.337085 11804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:47.337296 11804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:47.350236 11804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:47.350412 11804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:47.350637 11804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:08:47.367034 11804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:34731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:08:47.369069 11804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:47.370972 11804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:47.392513 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:47.392699 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:47.400192 11804 server_base.cc:1034] running on GCE node
W20260927 15:08:47.400552 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:47.401468 11804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:47.402915 11804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:47.404280 11804 hybrid_clock.cc:648] HybridClock initialized: now 1790521727404155 us; error 126 us; skew 500 ppm
I20260927 15:08:47.404731 11804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:08:47 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:47.408658 11804 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:08:47.414739 11804 webserver.cc:460] Webserver started at http://127.0.97.1:35217/ using document root <none> and password file <none>
I20260927 15:08:47.415678 11804 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:47.415861 11804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:47.416236 11804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:08:47 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:08:47.438107 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "389b34e02d2946edac51d3507a8a1634"
format_stamp: "Formatted at 2026-09-27 15:08:47 on dist-test-slave-g0w1"
server_key: "5e84da497491279e1c05f298a54ffa84"
server_key_iv: "1f85e9c57266215d71010b3484adc06e"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:47.439472 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "389b34e02d2946edac51d3507a8a1634"
format_stamp: "Formatted at 2026-09-27 15:08:47 on dist-test-slave-g0w1"
server_key: "5e84da497491279e1c05f298a54ffa84"
server_key_iv: "1f85e9c57266215d71010b3484adc06e"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:47.448392 11804 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 15:08:47.488399 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:47.490887 11804 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.004s
I20260927 15:08:47.491263 11804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "389b34e02d2946edac51d3507a8a1634"
format_stamp: "Formatted at 2026-09-27 15:08:47 on dist-test-slave-g0w1"
server_key: "5e84da497491279e1c05f298a54ffa84"
server_key_iv: "1f85e9c57266215d71010b3484adc06e"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:47.491677 11804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:47.540905 11804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:47.547254 11804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:47.547709 11804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:47.555289 11804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:47.563829 11804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:47.564046 11804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:08:47.564275 11804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:47.564416 11804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:08:47 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:08:47.952273 11804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40511
I20260927 15:08:47.952288 11936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40511 every 8 connection(s)
I20260927 15:08:47.954159 11804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:08:47.965322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 11804
I20260927 15:08:47.965596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:08:47.966602 388 external_mini_cluster.cc:1428] Setting key 74aef0635ebb0db4362fd8b28f65d0ae
Sep 27 15:08:47 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:08:48.002928 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:47.964950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60881 (local address 127.0.97.62:43023)
0927 15:08:47.973710 (+ 8760us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:47.973731 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:47.976482 (+ 2751us) server_negotiation.cc:407] Connection header received
0927 15:08:47.982102 (+ 5620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:47.982111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:47.982238 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:47.982323 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:47.984847 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:47.986089 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:47.988776 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:47.989481 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:47.989599 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:47.994935 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:47.994943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:47.994948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:47.994993 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:47.998813 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:48.000025 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:48.000031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:48.000034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:48.000191 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:48.000777 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:48.000782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:48.000785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:48.001123 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:48.001258 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:48.002301 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0927 15:08:48.002553 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8573,"thread_start_us":14266,"threads_started":1}
I20260927 15:08:48.005800 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:47.962394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43023 (local address 127.0.97.1:60881)
0927 15:08:47.976261 (+ 13867us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:47.976337 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:47.981684 (+ 5347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:47.982956 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:47.982969 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:47.983692 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:47.984594 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:47.984622 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:47.986352 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:47.986358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:47.988546 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:47.988567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:47.990030 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:47.990038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:47.990407 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:47.991315 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:47.991347 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:47.994609 (+ 3262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:47.999177 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:47.999186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:47.999205 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:47.999870 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:48.000355 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:48.000359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:48.000363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:48.000636 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:48.001359 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:48.001368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:48.001839 (+ 471us) client_negotiation.cc:769] Sending connection context
0927 15:08:48.003450 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0927 15:08:48.004122 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13490,"thread_start_us":16570,"threads_started":1}
I20260927 15:08:48.008241 11937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43023
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:08:48.008632 11937 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:48.009797 11937 heartbeater.cc:511] Master 127.0.97.62:43023 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 15:08:48.012984 11724 ts_manager.cc:194] Registered new tserver with Master: 389b34e02d2946edac51d3507a8a1634 (127.0.97.1:40511)
I20260927 15:08:48.015195 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:08:48.047474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43023
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:08:48 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 15:08:48.550985 11943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:48.551421 11943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:48.551587 11943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:48.559463 11943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:48.559625 11943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:48.559827 11943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:08:48.570767 11943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:34731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:08:48.572355 11943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:48.573763 11943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:48.591952 11957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:48.593216 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:48.593464 11960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:48.593958 11943 server_base.cc:1034] running on GCE node
I20260927 15:08:48.594830 11943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:48.596184 11943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:48.597414 11943 hybrid_clock.cc:648] HybridClock initialized: now 1790521728597339 us; error 65 us; skew 500 ppm
I20260927 15:08:48.597846 11943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:08:48 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:48.601989 11943 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:08:48.603771 11943 webserver.cc:460] Webserver started at http://127.0.97.2:40657/ using document root <none> and password file <none>
I20260927 15:08:48.604511 11943 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:48.604642 11943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:48.604980 11943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:08:48 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:08:48.623534 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "53853da75b6347b4aa773184cab4b477"
format_stamp: "Formatted at 2026-09-27 15:08:48 on dist-test-slave-g0w1"
server_key: "6845e64ebe78f671a79a73a37705eb4f"
server_key_iv: "471e3802459671f0df93a1ea76ff21ce"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:48.624982 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "53853da75b6347b4aa773184cab4b477"
format_stamp: "Formatted at 2026-09-27 15:08:48 on dist-test-slave-g0w1"
server_key: "6845e64ebe78f671a79a73a37705eb4f"
server_key_iv: "471e3802459671f0df93a1ea76ff21ce"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:48.633270 11943 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260927 15:08:48.654263 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:48.656425 11943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:08:48.656648 11943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "53853da75b6347b4aa773184cab4b477"
format_stamp: "Formatted at 2026-09-27 15:08:48 on dist-test-slave-g0w1"
server_key: "6845e64ebe78f671a79a73a37705eb4f"
server_key_iv: "471e3802459671f0df93a1ea76ff21ce"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:48.656915 11943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:48.685771 11943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:48.691560 11943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:48.691917 11943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:48.693181 11943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:48.695205 11943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:48.695318 11943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:48.695503 11943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:48.695570 11943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:48.750110 11943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45453
I20260927 15:08:48.750113 12079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45453 every 8 connection(s)
I20260927 15:08:48.752053 11943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:08:48.752976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 11943
I20260927 15:08:48.753147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:08:48.753647 388 external_mini_cluster.cc:1428] Setting key 426fcc649452dc5b8db059895d2fc165
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Sep 27 15:08:48 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:08:48.807957 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:48.757880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43023 (local address 127.0.97.2:50605)
0927 15:08:48.761881 (+ 4001us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:48.761952 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:48.765193 (+ 3241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:48.780359 (+ 15166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:48.780380 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:48.781252 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:48.782287 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:48.782322 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:48.784299 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:48.784308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:48.787444 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:48.787489 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:48.789078 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:48.789086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:48.789482 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:48.792735 (+ 3253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:48.792768 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:48.797446 (+ 4678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:48.802275 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:48.802289 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:48.802315 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:48.803097 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:48.803670 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:48.803676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:48.803681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:48.803971 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:48.804809 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:48.804821 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:48.805306 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 15:08:48.805622 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 15:08:48.806260 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3646,"thread_start_us":3403,"threads_started":1}
I20260927 15:08:48.809368 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:48.763304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50605 (local address 127.0.97.62:43023)
0927 15:08:48.779544 (+ 16240us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:48.779553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:48.779586 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:08:48.779661 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:48.779670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:48.779762 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:48.782546 (+ 2784us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:48.782749 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:48.784057 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:48.787797 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:48.788543 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:48.788719 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:48.797840 (+ 9121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:48.797850 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:48.797858 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:48.797912 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:48.802035 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:48.803286 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:48.803296 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:48.803300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:48.803507 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:48.804123 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:48.804130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:48.804135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:48.804542 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:48.804716 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:48.808548 (+ 3832us) server_negotiation.cc:299] Negotiation successful
0927 15:08:48.808889 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16060,"thread_start_us":19769,"threads_started":1}
I20260927 15:08:48.811888 12080 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43023
I20260927 15:08:48.812232 12080 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:48.813158 12080 heartbeater.cc:511] Master 127.0.97.62:43023 requested a full tablet report, sending...
I20260927 15:08:48.815127 11724 ts_manager.cc:194] Registered new tserver with Master: 53853da75b6347b4aa773184cab4b477 (127.0.97.2:45453)
I20260927 15:08:48.816428 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:08:48.830623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43023
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:08:49.019732 11937 heartbeater.cc:503] Master 127.0.97.62:43023 was elected leader, sending a full tablet report...
W20260927 15:08:49.198525 12086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:49.198939 12086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:49.199188 12086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:49.207302 12086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:49.207463 12086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:49.207645 12086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:08:49.218674 12086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:34731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:08:49.220203 12086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:49.221623 12086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:49.237088 12091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:49.237064 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:49.237315 12092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:49.238061 12086 server_base.cc:1034] running on GCE node
I20260927 15:08:49.238735 12086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:49.239957 12086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:49.241209 12086 hybrid_clock.cc:648] HybridClock initialized: now 1790521729241135 us; error 71 us; skew 500 ppm
I20260927 15:08:49.241564 12086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:08:49 dist-test-slave-g0w1 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:49.244966 12086 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:08:49.246515 12086 webserver.cc:460] Webserver started at http://127.0.97.3:43463/ using document root <none> and password file <none>
I20260927 15:08:49.247362 12086 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:49.247514 12086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:49.247839 12086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:08:49 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:08:49.263051 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "12f5ca1e320f4c58b99c60f054114ea1"
format_stamp: "Formatted at 2026-09-27 15:08:49 on dist-test-slave-g0w1"
server_key: "59802a475952deaa985643133e1e883c"
server_key_iv: "bee48222269c453547ca1574bef2b4bf"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:49.264253 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "12f5ca1e320f4c58b99c60f054114ea1"
format_stamp: "Formatted at 2026-09-27 15:08:49 on dist-test-slave-g0w1"
server_key: "59802a475952deaa985643133e1e883c"
server_key_iv: "bee48222269c453547ca1574bef2b4bf"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:49.270519 12086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260927 15:08:49.286571 12101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:49.288437 12086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:08:49.288651 12086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "12f5ca1e320f4c58b99c60f054114ea1"
format_stamp: "Formatted at 2026-09-27 15:08:49 on dist-test-slave-g0w1"
server_key: "59802a475952deaa985643133e1e883c"
server_key_iv: "bee48222269c453547ca1574bef2b4bf"
server_key_version: "kuduclusterkey@0"
I20260927 15:08:49.288905 12086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:49.311045 12086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:49.316291 12086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:49.316632 12086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:49.317664 12086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:49.319145 12086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:49.319264 12086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:49.319419 12086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:49.319494 12086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:49.359514 12086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41033
I20260927 15:08:49.359548 12213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41033 every 8 connection(s)
I20260927 15:08:49.360913 12086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:08:49.366447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 12086
I20260927 15:08:49.366647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:08:49.367146 388 external_mini_cluster.cc:1428] Setting key 73aa006d7378f480b27c69391434a216
Sep 27 15:08:49 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:08:49.383692 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.363821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37349 (local address 127.0.97.62:43023)
0927 15:08:49.364471 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.364476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.365615 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 15:08:49.367706 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.367712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.367857 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.367982 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:49.370135 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.370948 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.372787 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.373272 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.373379 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.377165 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:49.377171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:49.377175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:49.377218 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:49.379895 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.380759 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.380764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.380766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.380890 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.381438 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.381442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.381444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.381871 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:49.382022 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.383032 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.383371 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":379,"threads_started":1}
I20260927 15:08:49.384800 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.364593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43023 (local address 127.0.97.3:37349)
0927 15:08:49.365398 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:49.365448 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:49.367380 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:49.368400 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:49.368411 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:49.368967 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:49.369909 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.369938 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.371202 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.371207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.372586 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.372609 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.373507 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.373513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.373749 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.374561 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:49.374587 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:49.376880 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:49.380066 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:49.380076 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:49.380096 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:49.380589 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:49.381014 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:49.381019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:49.381023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:49.381279 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:49.382039 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:49.382047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:49.382532 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 15:08:49.382858 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 15:08:49.383513 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
I20260927 15:08:49.386322 12214 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43023
I20260927 15:08:49.386611 12214 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:49.387501 12214 heartbeater.cc:511] Master 127.0.97.62:43023 requested a full tablet report, sending...
I20260927 15:08:49.388975 11724 ts_manager.cc:194] Registered new tserver with Master: 12f5ca1e320f4c58b99c60f054114ea1 (127.0.97.3:41033)
I20260927 15:08:49.389773 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37349
I20260927 15:08:49.396282 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:08:49.417124 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.401965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.97.62:43023)
0927 15:08:49.402178 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.402184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.402650 (+ 466us) server_negotiation.cc:407] Connection header received
0927 15:08:49.402837 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.402842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.402929 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.403032 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:49.404803 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.405580 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.407421 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.407901 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.408012 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.410866 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:49.410872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:49.410875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:49.410915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:49.414065 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.414936 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.414940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.414943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.415150 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.415658 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.415663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.415665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.416008 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:49.416255 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.416697 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.416882 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:08:49.421867 11724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:08:49.429726 11724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:08:49.461833 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.443928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.97.3:41033)
0927 15:08:49.444988 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.445003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.446459 (+ 1456us) server_negotiation.cc:407] Connection header received
0927 15:08:49.448497 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.448511 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.448887 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.449100 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:49.450429 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.454221 (+ 3792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.459439 (+ 5218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.460440 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.460628 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.460886 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.461251 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.461445 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":858,"threads_started":1}
I20260927 15:08:49.462774 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.443641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41033 (local address 127.0.0.1:47244)
0927 15:08:49.446249 (+ 2608us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:49.446303 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:49.448165 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:49.449529 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:49.449537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:49.449572 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:49.450150 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.450169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.454524 (+ 4355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.454533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.459261 (+ 4728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.459272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.460673 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.460680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.460896 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.460995 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 15:08:49.462057 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0927 15:08:49.462386 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2233,"mutex_wait_us":1,"thread_start_us":13192,"threads_started":1}
I20260927 15:08:49.466884 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.443704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40511 (local address 127.0.0.1:41084)
0927 15:08:49.456106 (+ 12402us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:49.456133 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:49.456306 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:49.458388 (+ 2082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:49.458393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:49.458418 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:49.458942 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.458953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.460931 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.460938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.464010 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.464028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.465953 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.465959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.466088 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.466115 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:08:49.466332 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:08:49.466486 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12220,"mutex_wait_us":33,"thread_start_us":2638,"threads_started":1}
I20260927 15:08:49.466919 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.444613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.0.97.1:40511)
0927 15:08:49.457703 (+ 13090us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.457710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.457732 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:08:49.457811 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.457816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.457987 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.458222 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:49.459160 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.460738 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.464215 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.465702 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.465845 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.466068 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.466390 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.466553 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12929,"thread_start_us":13810,"threads_started":1}
I20260927 15:08:49.467336 12149 tablet_service.cc:1467] Processing CreateTablet for tablet fb3327dae6de42498cdb2bc30b68c9e6 (DEFAULT_TABLE table=test-table [id=083f8b7fe068466a843659853effa2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:08:49.469031 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.447443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45453 (local address 127.0.0.1:45578)
0927 15:08:49.457194 (+ 9751us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:49.457219 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:49.457378 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:49.458140 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:49.458146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:49.458176 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:49.458684 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.458698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.460703 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.460708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.464013 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.464027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.468145 (+ 4118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.468152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.468340 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.468378 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:08:49.468468 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:08:49.468627 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9577,"mutex_wait_us":101,"thread_start_us":8192,"threads_started":1}
I20260927 15:08:49.469060 12149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3327dae6de42498cdb2bc30b68c9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:49.469383 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.448068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.0.97.2:45453)
0927 15:08:49.455440 (+ 7372us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.455455 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.457366 (+ 1911us) server_negotiation.cc:407] Connection header received
0927 15:08:49.457564 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.457573 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.457896 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.458129 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:49.458887 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.460533 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.466180 (+ 5647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.467677 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.467811 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.468828 (+ 1017us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.468902 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.469073 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7174,"thread_start_us":7045,"threads_started":1}
I20260927 15:08:49.471621 11872 tablet_service.cc:1467] Processing CreateTablet for tablet fb3327dae6de42498cdb2bc30b68c9e6 (DEFAULT_TABLE table=test-table [id=083f8b7fe068466a843659853effa2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:08:49.473213 11872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3327dae6de42498cdb2bc30b68c9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:49.473354 12015 tablet_service.cc:1467] Processing CreateTablet for tablet fb3327dae6de42498cdb2bc30b68c9e6 (DEFAULT_TABLE table=test-table [id=083f8b7fe068466a843659853effa2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:08:49.474807 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3327dae6de42498cdb2bc30b68c9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:49.481711 12233 tablet_bootstrap.cc:492] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Bootstrap starting.
I20260927 15:08:49.486238 12233 tablet_bootstrap.cc:654] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:49.487392 12233 log.cc:824] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:49.490222 12236 tablet_bootstrap.cc:492] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Bootstrap starting.
I20260927 15:08:49.491384 12233 tablet_bootstrap.cc:492] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: No bootstrap required, opened a new log
I20260927 15:08:49.491349 12235 tablet_bootstrap.cc:492] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Bootstrap starting.
I20260927 15:08:49.491686 12233 ts_tablet_manager.cc:1397] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 15:08:49.494437 12236 tablet_bootstrap.cc:654] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:49.495424 12235 tablet_bootstrap.cc:654] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:49.495688 12236 log.cc:824] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:49.496685 12235 log.cc:824] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:49.501169 12235 tablet_bootstrap.cc:492] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: No bootstrap required, opened a new log
I20260927 15:08:49.501570 12235 ts_tablet_manager.cc:1397] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260927 15:08:49.503755 12236 tablet_bootstrap.cc:492] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: No bootstrap required, opened a new log
I20260927 15:08:49.504055 12236 ts_tablet_manager.cc:1397] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260927 15:08:49.504379 12233 raft_consensus.cc:348] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.504871 12233 raft_consensus.cc:374] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:49.505015 12233 raft_consensus.cc:729] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f5ca1e320f4c58b99c60f054114ea1, State: Initialized, Role: FOLLOWER
I20260927 15:08:49.505692 12233 consensus_queue.cc:260] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.506906 12214 heartbeater.cc:503] Master 127.0.97.62:43023 was elected leader, sending a full tablet report...
I20260927 15:08:49.507854 12233 ts_tablet_manager.cc:1428] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 15:08:49.512903 12235 raft_consensus.cc:348] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.513515 12235 raft_consensus.cc:374] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:49.513641 12235 raft_consensus.cc:729] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389b34e02d2946edac51d3507a8a1634, State: Initialized, Role: FOLLOWER
I20260927 15:08:49.514384 12235 consensus_queue.cc:260] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.515447 12236 raft_consensus.cc:348] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.515960 12236 raft_consensus.cc:374] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:49.516075 12236 raft_consensus.cc:729] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53853da75b6347b4aa773184cab4b477, State: Initialized, Role: FOLLOWER
I20260927 15:08:49.516572 12235 ts_tablet_manager.cc:1428] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260927 15:08:49.516752 12236 consensus_queue.cc:260] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.518143 12080 heartbeater.cc:503] Master 127.0.97.62:43023 was elected leader, sending a full tablet report...
I20260927 15:08:49.518889 12236 ts_tablet_manager.cc:1428] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260927 15:08:49.545459 12081 tablet.cc:2369] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:08:49.580389 12240 raft_consensus.cc:482] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:08:49.580796 12240 raft_consensus.cc:504] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.582636 12240 leader_election.cc:288] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12f5ca1e320f4c58b99c60f054114ea1 (127.0.97.3:41033), 53853da75b6347b4aa773184cab4b477 (127.0.97.2:45453)
I20260927 15:08:49.593446 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.583206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34961 (local address 127.0.97.3:41033)
0927 15:08:49.583456 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.583462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.585049 (+ 1587us) server_negotiation.cc:407] Connection header received
0927 15:08:49.585241 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.585245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.585351 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.585444 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:49.586077 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.587312 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.590660 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.592018 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.592157 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.592332 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.592986 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.593164 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:08:49.593434 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.583206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34179 (local address 127.0.97.2:45453)
0927 15:08:49.583565 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.583575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.585105 (+ 1530us) server_negotiation.cc:407] Connection header received
0927 15:08:49.585606 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.585610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.585677 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.585746 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:49.586441 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.587467 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.590660 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.592000 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.592157 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.592262 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.593000 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.593164 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260927 15:08:49.593734 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.583130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45453 (local address 127.0.97.1:34179)
0927 15:08:49.584913 (+ 1783us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:49.584929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:49.585099 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:49.585805 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:49.585813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:49.585847 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:49.586255 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.586266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.587620 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.587625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.590418 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.590440 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.592441 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.592453 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.592716 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.592791 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 15:08:49.592901 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:08:49.593132 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"mutex_wait_us":301,"thread_start_us":1369,"threads_started":1}
I20260927 15:08:49.593858 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.583010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41033 (local address 127.0.97.1:34961)
0927 15:08:49.584885 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:49.584911 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:49.585107 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:49.585480 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:49.585484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:49.585505 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:49.585863 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.585877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.587498 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.587503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.590418 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:49.590440 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.592483 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.592489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:49.592722 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.592786 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 15:08:49.592901 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:08:49.593105 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"mutex_wait_us":189,"spinlock_wait_cycles":11776,"thread_start_us":1477,"threads_started":1}
I20260927 15:08:49.594906 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3327dae6de42498cdb2bc30b68c9e6" candidate_uuid: "389b34e02d2946edac51d3507a8a1634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53853da75b6347b4aa773184cab4b477" is_pre_election: true
I20260927 15:08:49.594910 12169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3327dae6de42498cdb2bc30b68c9e6" candidate_uuid: "389b34e02d2946edac51d3507a8a1634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f5ca1e320f4c58b99c60f054114ea1" is_pre_election: true
I20260927 15:08:49.595930 12035 raft_consensus.cc:2454] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 389b34e02d2946edac51d3507a8a1634 in term 0.
I20260927 15:08:49.595539 12169 raft_consensus.cc:2454] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 389b34e02d2946edac51d3507a8a1634 in term 0.
I20260927 15:08:49.596748 11827 leader_election.cc:302] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12f5ca1e320f4c58b99c60f054114ea1, 389b34e02d2946edac51d3507a8a1634, 53853da75b6347b4aa773184cab4b477; no voters:
I20260927 15:08:49.597223 12240 raft_consensus.cc:2781] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:08:49.597385 12240 raft_consensus.cc:482] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:08:49.597492 12240 raft_consensus.cc:3037] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:49.600207 12240 raft_consensus.cc:504] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
I20260927 15:08:49.600893 12240 leader_election.cc:288] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [CANDIDATE]: Term 1 election: Requested vote from peers 12f5ca1e320f4c58b99c60f054114ea1 (127.0.97.3:41033), 53853da75b6347b4aa773184cab4b477 (127.0.97.2:45453)
I20260927 15:08:49.601513 12169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3327dae6de42498cdb2bc30b68c9e6" candidate_uuid: "389b34e02d2946edac51d3507a8a1634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f5ca1e320f4c58b99c60f054114ea1"
I20260927 15:08:49.601733 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3327dae6de42498cdb2bc30b68c9e6" candidate_uuid: "389b34e02d2946edac51d3507a8a1634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53853da75b6347b4aa773184cab4b477"
I20260927 15:08:49.601794 12169 raft_consensus.cc:3037] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:49.601982 12035 raft_consensus.cc:3037] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:49.605293 12035 raft_consensus.cc:2454] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 389b34e02d2946edac51d3507a8a1634 in term 1.
I20260927 15:08:49.605291 12169 raft_consensus.cc:2454] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 389b34e02d2946edac51d3507a8a1634 in term 1.
I20260927 15:08:49.605801 11827 leader_election.cc:302] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12f5ca1e320f4c58b99c60f054114ea1, 389b34e02d2946edac51d3507a8a1634, 53853da75b6347b4aa773184cab4b477; no voters:
I20260927 15:08:49.606226 12240 raft_consensus.cc:2781] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:08:49.606734 12240 raft_consensus.cc:686] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 1 LEADER]: Becoming Leader. State: Replica: 389b34e02d2946edac51d3507a8a1634, State: Running, Role: LEADER
I20260927 15:08:49.607360 12240 consensus_queue.cc:237] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } }
W20260927 15:08:49.612587 12215 tablet.cc:2369] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:08:49.612562 11723 catalog_manager.cc:5495] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 reported cstate change: term changed from 0 to 1, leader changed from <none> to 389b34e02d2946edac51d3507a8a1634 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "389b34e02d2946edac51d3507a8a1634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "389b34e02d2946edac51d3507a8a1634" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 } health_report { overall_health: UNKNOWN } } }
I20260927 15:08:49.653147 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.647477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.0.97.1:40511)
0927 15:08:49.647660 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.647665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.647683 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:08:49.647889 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.647893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.647973 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.648112 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:08:49.648507 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.649224 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.650742 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.651316 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.651426 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.651991 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:08:49.652554 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:08:49.652682 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.652852 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.652933 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:08:49.662513 12035 raft_consensus.cc:1261] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 1 FOLLOWER]: Refusing update from remote peer 389b34e02d2946edac51d3507a8a1634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:08:49.662513 12169 raft_consensus.cc:1261] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 1 FOLLOWER]: Refusing update from remote peer 389b34e02d2946edac51d3507a8a1634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:08:49.663779 12244 consensus_queue.cc:1035] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53853da75b6347b4aa773184cab4b477" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:08:49.664181 12240 consensus_queue.cc:1035] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f5ca1e320f4c58b99c60f054114ea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:08:49.677314 12248 mvcc.cc:205] Tried to move back new op lower bound from 7333977004685574144 to 7333977004475736064. Current Snapshot: MvccSnapshot[applied={T|T < 7333977004685574144}]
I20260927 15:08:49.678211 12247 mvcc.cc:205] Tried to move back new op lower bound from 7333977004685574144 to 7333977004475736064. Current Snapshot: MvccSnapshot[applied={T|T < 7333977004685574144}]
I20260927 15:08:49.697227 11724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:08:49.697616 11724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:08:49.701759 11724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b2a84a2cff47e4bbf152692b35b748: Sending DeleteTablet for 3 replicas of tablet fb3327dae6de42498cdb2bc30b68c9e6
I20260927 15:08:49.703045 12149 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3327dae6de42498cdb2bc30b68c9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:08:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47244
I20260927 15:08:49.703218 12015 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3327dae6de42498cdb2bc30b68c9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:08:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45578
I20260927 15:08:49.703923 11872 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3327dae6de42498cdb2bc30b68c9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:08:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41084
I20260927 15:08:49.704303 12257 tablet_replica.cc:330] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: stopping tablet replica
I20260927 15:08:49.704867 12259 tablet_replica.cc:330] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: stopping tablet replica
I20260927 15:08:49.705412 12259 raft_consensus.cc:2229] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:08:49.705832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 11804
I20260927 15:08:49.706414 12259 raft_consensus.cc:2258] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:08:49.708652 12259 ts_tablet_manager.cc:1905] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:08:49.715155 12257 raft_consensus.cc:2229] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:08:49.716521 12257 raft_consensus.cc:2258] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:08:49.717442 12258 tablet_replica.cc:330] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: stopping tablet replica
I20260927 15:08:49.718055 12258 raft_consensus.cc:2229] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:08:49.718647 12258 raft_consensus.cc:2258] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:08:49.718959 12257 ts_tablet_manager.cc:1905] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:08:49.721124 12258 ts_tablet_manager.cc:1905] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:08:49.726317 12257 ts_tablet_manager.cc:1918] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:08:49.726585 12257 log.cc:1196] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/fb3327dae6de42498cdb2bc30b68c9e6
I20260927 15:08:49.727206 12257 ts_tablet_manager.cc:1939] T fb3327dae6de42498cdb2bc30b68c9e6 P 12f5ca1e320f4c58b99c60f054114ea1: Deleting consensus metadata
I20260927 15:08:49.728492 12258 ts_tablet_manager.cc:1918] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:08:49.728821 12258 log.cc:1196] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/fb3327dae6de42498cdb2bc30b68c9e6
I20260927 15:08:49.729445 12258 ts_tablet_manager.cc:1939] T fb3327dae6de42498cdb2bc30b68c9e6 P 53853da75b6347b4aa773184cab4b477: Deleting consensus metadata
I20260927 15:08:49.729465 11711 catalog_manager.cc:4841] TS 12f5ca1e320f4c58b99c60f054114ea1 (127.0.97.3:41033): tablet fb3327dae6de42498cdb2bc30b68c9e6 (table test-table [id=083f8b7fe068466a843659853effa2eb]) successfully deleted
Sep 27 15:08:49 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:08:49.731012 12259 ts_tablet_manager.cc:1918] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:08:49.731376 12259 log.cc:1196] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/fb3327dae6de42498cdb2bc30b68c9e6
I20260927 15:08:49.731595 11710 catalog_manager.cc:4841] TS 53853da75b6347b4aa773184cab4b477 (127.0.97.2:45453): tablet fb3327dae6de42498cdb2bc30b68c9e6 (table test-table [id=083f8b7fe068466a843659853effa2eb]) successfully deleted
I20260927 15:08:49.732069 12259 ts_tablet_manager.cc:1939] T fb3327dae6de42498cdb2bc30b68c9e6 P 389b34e02d2946edac51d3507a8a1634: Deleting consensus metadata
I20260927 15:08:49.734105 11710 catalog_manager.cc:4841] TS 389b34e02d2946edac51d3507a8a1634 (127.0.97.1:40511): tablet fb3327dae6de42498cdb2bc30b68c9e6 (table test-table [id=083f8b7fe068466a843659853effa2eb]) successfully deleted
I20260927 15:08:49.738489 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.708731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.0.97.1:40511)
0927 15:08:49.708903 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.708910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.715633 (+ 6723us) server_negotiation.cc:407] Connection header received
0927 15:08:49.715847 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.715853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.715945 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.716046 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:49.717889 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.719168 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.721464 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.722189 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.722298 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.731043 (+ 8745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:49.731111 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:49.731131 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:49.731196 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:49.735319 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.736195 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.736202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.736204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.736349 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.736786 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.736791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.736794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.737253 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:49.737575 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.737885 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.738129 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:08:49.739281 11932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:08:49.841572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 11804
I20260927 15:08:49.860740 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 11943
Sep 27 15:08:49 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:08:49.876953 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.861587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.97.2:45453)
0927 15:08:49.861759 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.861763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.861782 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:08:49.861971 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.861975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.862055 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.862146 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:49.863483 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.864375 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.865681 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.866067 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.866163 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:49.870536 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:49.870553 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:49.870569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:49.870606 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:49.873882 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.874725 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.874734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.874736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.874885 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:49.875437 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:49.875440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:49.875443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:49.875841 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:49.876055 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:49.876485 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 15:08:49.876665 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:08:49.877740 12075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:08:49.973266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 11943
I20260927 15:08:49.994711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 12086
Sep 27 15:08:50 dist-test-slave-g0w1 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:08:50.017750 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:49.995571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.97.3:41033)
0927 15:08:49.995763 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:49.995770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:49.995791 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:08:49.995954 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:49.995958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:49.996039 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:49.996126 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:49.997447 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:49.998207 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:49.999687 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:50.000091 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:50.000193 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:50.010891 (+ 10698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:50.010909 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:50.010925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:50.010976 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:50.014376 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:50.015584 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:50.015592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:50.015595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:50.015753 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:50.016155 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:50.016158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:50.016160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:50.016550 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:50.016781 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:50.017140 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 15:08:50.017364 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:08:50.018529 12209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:08:50.109335 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 12086
I20260927 15:08:50.127036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 11692
I20260927 15:08:50.128477 11754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:08:50.241988 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 11692
I20260927 15:08:50.266290 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:08:50.602634 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:08:50.602900 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:08:50.949656 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:08:50.950 UTC [11278] LOG: received smart shutdown request
2026-09-27 15:08:50.953 UTC [11278] LOG: background worker "logical replication launcher" (PID 11287) exited with exit code 1
2026-09-27 15:08:50.954 UTC [11282] LOG: shutting down
2026-09-27 15:08:51.149 UTC [11278] LOG: database system is shut down
2026-09-27T15:08:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53048 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:08:51 dist-test-slave-g0w1 krb5kdc[12273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:08:51 dist-test-slave-g0w1 krb5kdc[12273](info): set up 2 sockets
Sep 27 15:08:51 dist-test-slave-g0w1 krb5kdc[12273](info): commencing operation
krb5kdc: starting...
W20260927 15:08:52.467587 388 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:08:52 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:08:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 15:08:52.822096 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0r1zNn/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/postgres -l logfile start
2026-09-27 15:08:55.592 UTC [12303] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:08:55.592 UTC [12303] LOG: listening on IPv4 address "127.0.97.20", port 34081
2026-09-27 15:08:55.596 UTC [12303] LOG: listening on Unix socket "/tmp/.s.PGSQL.34081"
2026-09-27 15:08:55.621 UTC [12306] LOG: database system was shut down at 2026-09-27 15:08:53 UTC
2026-09-27 15:08:55.625 UTC [12303] LOG: database system is ready to accept connections
I20260927 15:08:56.607664 388 mini_postgres.cc:96] Postgres bound to 34081
2026-09-27 15:08:56.626 UTC [12314] FATAL: database "slave" does not exist
127.0.97.20:34081 - accepting connections
I20260927 15:08:56.627154 388 mini_ranger.cc:161] Starting Ranger...
I20260927 15:08:56.661654 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:08:57.234460 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:08:57.234577 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-admin
2026-09-27 15:08:57,563 [I] DB FLAVOR :POSTGRES
2026-09-27 15:08:57,564 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:08:57,564 [I] Checking connection..
2026-09-27 15:08:57,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:08:57,853 [I] Checking connection passed.
2026-09-27 15:08:57,854 [I] --------- Verifying version history table ---------
2026-09-27 15:08:57,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:58,111 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:08:58,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:58,380 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:08:58,380 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:08:58,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:08:58.637 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 15:08:58,644 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:08:58,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:58,896 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:08:58,896 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:08:58,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:08:59,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:08:59,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g0w1.c.gcp-upstream.internal','N') ;"
2026-09-27 15:08:59,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:08:59,906 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:08:59,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:09:00,186 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:09:00,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:09:00,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:09:00,721 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:09:00,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:09:01.895 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:01.914 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:01.935 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:01.954 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:01.975 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:01.996 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.064 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.072 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.089 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.102 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.115 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.133 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.144 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.152 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.160 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:02.168 UTC [12486] WARNING: there is no transaction in progress
2026-09-27 15:09:03,018 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:09:03,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:09:03,270 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:09:03,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:09:03,538 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:09:03,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:09:03,791 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:09:03,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:09:04,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:09:04,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g0w1.c.gcp-upstream.internal';"
2026-09-27 15:09:04,587 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:09:04,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:09:04,849 [I] DB_PATCHES have already been applied
I20260927 15:09:04.856886 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0r1zNn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:09:04.857046 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 15:09:04.872390 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:60913
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:09:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:09:06.041772 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.415s user 0.011s sys 0.036s
Sep 27, 2026 3:09:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:09:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:09:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:09:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:09:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:09:29.507251 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:09:29 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521769, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:09:29.673085 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:09:29.694705 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:09:30.694666 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:09:30.694780 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms
I20260927 15:09:30.694830 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34081
2026-09-27 15:09:31,056 [I] DB FLAVOR :POSTGRES
2026-09-27 15:09:31,056 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:09:31,056 [I] Checking connection
2026-09-27 15:09:31,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:09:31,308 [I] connection success
2026-09-27 15:09:31,309 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:09:31,309 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:09:31,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:09:31,598 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:09:31,598 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:09:31,599 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:09:31,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:09:31,903 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:09:32.481595 388 mini_ranger_kms.cc:310] Created kms service
I20260927 15:09:32.718498 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:09:32.770079 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:09:32 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521769, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:33.196900 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:09:33.197276 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task0r1zNn/build/asan/bin/postgresql.jar:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0r1zNn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:09:33.197350 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 15:09:33.215827 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12667
I20260927 15:09:33.215991 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:33055
Sep 27, 2026 3:09:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:09:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:09:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:09:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:09:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:09:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 15:09:35 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521775, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:35.595865 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.923s user 0.081s sys 0.291s
I20260927 15:09:35.596134 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:09:35.596315 388 mini_ranger_kms.cc:401] 127.0.97.20:33055/kms/v1/keys
Sep 27 15:09:35 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521775, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 15:09:36 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521775, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 15:09:38 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:09:38.943594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37197
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37197
--ranger_config_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:39.294615 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:39.294956 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:39.295048 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:39.302820 12717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:39.302956 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:39.303036 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:39.303153 12717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:39.303267 12717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:39.314160 12717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:33055/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37197
--ranger_config_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:39.315677 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:39.317200 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:39.330498 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:39.330605 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:39.330530 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:39.331115 12717 server_base.cc:1034] running on GCE node
I20260927 15:09:39.332118 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:39.333784 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:39.335127 12717 hybrid_clock.cc:648] HybridClock initialized: now 1790521779335050 us; error 60 us; skew 500 ppm
I20260927 15:09:39.335557 12717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:39 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:39.339368 12717 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:09:39.341264 12717 webserver.cc:460] Webserver started at http://127.0.97.62:42563/ using document root <none> and password file <none>
I20260927 15:09:39.342068 12717 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:39.342233 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:39.342582 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:39 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:39.397532 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb"
format_stamp: "Formatted at 2026-09-27 15:09:39 on dist-test-slave-g0w1"
server_key: "82b99b24332d77aeb607fc498e1dc0f9"
server_key_iv: "3a69bc65ced30fb4c8241e9763fd919a"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:39.398521 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb"
format_stamp: "Formatted at 2026-09-27 15:09:39 on dist-test-slave-g0w1"
server_key: "82b99b24332d77aeb607fc498e1dc0f9"
server_key_iv: "3a69bc65ced30fb4c8241e9763fd919a"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:39.404624 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:09:39.426510 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:39.428607 12717 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:09:39.428902 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb"
format_stamp: "Formatted at 2026-09-27 15:09:39 on dist-test-slave-g0w1"
server_key: "82b99b24332d77aeb607fc498e1dc0f9"
server_key_iv: "3a69bc65ced30fb4c8241e9763fd919a"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:39.429296 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:39.455711 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:39.460426 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:39.460805 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:39.481745 12717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37197
I20260927 15:09:39.481774 12783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37197 every 8 connection(s)
I20260927 15:09:39.483301 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:39.487679 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:39.490872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 12717
I20260927 15:09:39.491230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:09:39.491899 388 external_mini_cluster.cc:1428] Setting key a893b10e19075d849c2dd663a437ead3
I20260927 15:09:39.501870 12784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Bootstrap starting.
I20260927 15:09:39.506240 12784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:39.507895 12784 log.cc:824] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:09:39 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:39.511828 12784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: No bootstrap required, opened a new log
I20260927 15:09:39.520190 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:39.494430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.62:37197)
0927 15:09:39.495406 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:39.495420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:39.495466 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:09:39.497206 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:39.497235 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:39.497736 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:39.498182 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:39.500074 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:39.501552 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:39.503651 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:39.504497 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:39.504669 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:39.510206 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:39.510216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:39.510234 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:39.510281 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:39.514505 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:39.515595 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:39.515606 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:39.515617 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:39.515864 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:39.516319 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:39.516324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:39.516328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:39.517186 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:39.517552 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:39.518034 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 15:09:39.518551 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":437,"threads_started":1}
I20260927 15:09:39.523204 12784 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:39.523764 12784 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:39.523972 12784 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cec74718e44be7a78c8c4e8a5f8ddb, State: Initialized, Role: FOLLOWER
I20260927 15:09:39.524693 12784 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:39.524973 12784 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:39.525130 12784 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:39.525316 12784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:39.528081 12784 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:39.528825 12784 leader_election.cc:302] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0cec74718e44be7a78c8c4e8a5f8ddb; no voters:
I20260927 15:09:39.529627 12784 leader_election.cc:288] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:39.529783 12789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:39.532297 12789 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 LEADER]: Becoming Leader. State: Replica: d0cec74718e44be7a78c8c4e8a5f8ddb, State: Running, Role: LEADER
I20260927 15:09:39.532913 12789 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:39.533131 12784 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:39.536151 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0cec74718e44be7a78c8c4e8a5f8ddb. Latest consensus state: current_term: 1 leader_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } } }
I20260927 15:09:39.536725 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:39.537348 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } } }
I20260927 15:09:39.537668 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:39.537670 12797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:39.543380 12797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:39.551946 12784 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:09:39.553148 12797 catalog_manager.cc:1347] Generated new cluster ID: fc9047d8bfe04431a1089ce5a365a099
I20260927 15:09:39.553261 12797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:39.577577 12797 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:39.579954 12797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:39.596644 12797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Generated new TSK 0
I20260927 15:09:39.597941 12797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:09:40.354923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37197
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:40 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:09:40.806169 12829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:40.806754 12829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:40.806941 12829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:40.819135 12829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:40.819311 12829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:40.819510 12829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:09:40.831553 12829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:33055/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:40.833237 12829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:40.834987 12829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:40.855767 12839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:40.856060 12840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:40.858752 12829 server_base.cc:1034] running on GCE node
W20260927 15:09:40.859261 12842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:40.860087 12829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:40.862548 12829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:40.863858 12829 hybrid_clock.cc:648] HybridClock initialized: now 1790521780863770 us; error 76 us; skew 500 ppm
I20260927 15:09:40.864243 12829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:40 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:40.874845 12829 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:09:40.877048 12829 webserver.cc:460] Webserver started at http://127.0.97.1:41839/ using document root <none> and password file <none>
I20260927 15:09:40.878091 12829 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:40.878245 12829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:40.878599 12829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:40 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:40.897518 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "4b7cdba28b67416fb879b6010914cabc"
format_stamp: "Formatted at 2026-09-27 15:09:40 on dist-test-slave-g0w1"
server_key: "821e9b3c0d53a0cc185858524de440a6"
server_key_iv: "ba738b9eb8dcdd17297197cfdbe767b4"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:40.898455 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4b7cdba28b67416fb879b6010914cabc"
format_stamp: "Formatted at 2026-09-27 15:09:40 on dist-test-slave-g0w1"
server_key: "821e9b3c0d53a0cc185858524de440a6"
server_key_iv: "ba738b9eb8dcdd17297197cfdbe767b4"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:40.904892 12829 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260927 15:09:40.940439 12849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:40.942871 12829 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260927 15:09:40.943241 12829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "4b7cdba28b67416fb879b6010914cabc"
format_stamp: "Formatted at 2026-09-27 15:09:40 on dist-test-slave-g0w1"
server_key: "821e9b3c0d53a0cc185858524de440a6"
server_key_iv: "ba738b9eb8dcdd17297197cfdbe767b4"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:40.943657 12829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:40.986351 12829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:40.992261 12829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:40.992707 12829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:41.000269 12829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:41.008780 12829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:41.009013 12829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 15:09:41.009261 12829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:41.009393 12829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:09:41 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:41.388902 12829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43537
I20260927 15:09:41.388928 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43537 every 8 connection(s)
I20260927 15:09:41.390501 12829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:41.401089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 12829
I20260927 15:09:41.401362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:09:41.402365 388 external_mini_cluster.cc:1428] Setting key a834b11627798ae63272727867ce6a8c
Sep 27 15:09:41 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:41.433822 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:41.393962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33919 (local address 127.0.97.62:37197)
0927 15:09:41.406020 (+ 12058us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:41.406027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:41.410720 (+ 4693us) server_negotiation.cc:407] Connection header received
0927 15:09:41.410800 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:41.410806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:41.410919 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:41.411065 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:41.413335 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:41.414476 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:41.417213 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:41.417922 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:41.418071 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:41.423328 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:41.423336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:41.423342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:41.423389 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:41.427222 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:41.428453 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:41.428459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:41.428463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:41.428649 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:41.429257 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:41.429262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:41.429265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:41.429655 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:41.429807 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:41.433083 (+ 3276us) server_negotiation.cc:299] Negotiation successful
0927 15:09:41.433435 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11893,"thread_start_us":11691,"threads_started":1}
I20260927 15:09:41.434674 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:41.394647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37197 (local address 127.0.97.1:33919)
0927 15:09:41.406713 (+ 12066us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:41.406791 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:41.409389 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:41.411563 (+ 2174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:41.411577 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:41.412226 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:41.413110 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:41.413140 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:41.414708 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:41.414714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:41.417003 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:41.417020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:41.418437 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:41.418446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:41.418775 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:41.419845 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:41.419885 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:41.422919 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:41.427475 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:41.427487 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:41.427509 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:41.428229 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:41.428806 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:41.428812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:41.428816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:41.429104 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:41.429885 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:41.429895 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:41.430364 (+ 469us) client_negotiation.cc:769] Sending connection context
0927 15:09:41.430705 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 15:09:41.431335 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11677,"thread_start_us":15098,"threads_started":1}
I20260927 15:09:41.436892 12962 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37197
I20260927 15:09:41.437233 12962 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:41.438133 12962 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:41.440834 12749 ts_manager.cc:194] Registered new tserver with Master: 4b7cdba28b67416fb879b6010914cabc (127.0.97.1:43537)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 15:09:41.443200 12749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:09:41.484982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37197
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:41 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 15:09:41.931735 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:41.932097 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:41.932261 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:41.940204 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:41.940387 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:41.940562 12969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:09:41.951645 12969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:33055/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:41.953584 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:41.955746 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:41.975595 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:41.975868 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:41.979357 12969 server_base.cc:1034] running on GCE node
W20260927 15:09:41.980464 12985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:41.981419 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:41.982738 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:41.983994 12969 hybrid_clock.cc:648] HybridClock initialized: now 1790521781983937 us; error 54 us; skew 500 ppm
I20260927 15:09:41.984396 12969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:41 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:41.988565 12969 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:09:41.990492 12969 webserver.cc:460] Webserver started at http://127.0.97.2:39039/ using document root <none> and password file <none>
I20260927 15:09:41.991515 12969 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:41.991657 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:41.991993 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:41 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:42.009402 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "57538487bcae4082a93e256ee9733b97"
format_stamp: "Formatted at 2026-09-27 15:09:41 on dist-test-slave-g0w1"
server_key: "72a427e2240dcbca79ab5e9829a1014d"
server_key_iv: "b6fac0231d85392e6b751a6efdfac455"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:42.010833 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "57538487bcae4082a93e256ee9733b97"
format_stamp: "Formatted at 2026-09-27 15:09:41 on dist-test-slave-g0w1"
server_key: "72a427e2240dcbca79ab5e9829a1014d"
server_key_iv: "b6fac0231d85392e6b751a6efdfac455"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:42.017071 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 15:09:42.036373 12992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:42.038596 12969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:09:42.038902 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "57538487bcae4082a93e256ee9733b97"
format_stamp: "Formatted at 2026-09-27 15:09:41 on dist-test-slave-g0w1"
server_key: "72a427e2240dcbca79ab5e9829a1014d"
server_key_iv: "b6fac0231d85392e6b751a6efdfac455"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:42.039256 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:42.070590 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:42.076190 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:42.076596 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:42.077862 12969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:42.079792 12969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:42.079931 12969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 15:09:42.080075 12969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:42.080152 12969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:42.119642 12969 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34027
I20260927 15:09:42.119666 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34027 every 8 connection(s)
I20260927 15:09:42.121419 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:42.125907 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 12969
I20260927 15:09:42.126209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:09:42.126766 388 external_mini_cluster.cc:1428] Setting key 588e0dc80e27e1e0538174b2038b2b67
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:09:42.159739 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.124499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58623 (local address 127.0.97.62:37197)
0927 15:09:42.131517 (+ 7018us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:42.131527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:42.135140 (+ 3613us) server_negotiation.cc:407] Connection header received
0927 15:09:42.135266 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:42.135275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:42.135391 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:42.135539 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:42.141507 (+ 5968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.142649 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.145624 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.146436 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.146626 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.151634 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:42.151645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:42.151653 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:42.151719 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:42.155621 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.156798 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.156807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.156813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.156986 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.157634 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.157642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.157648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.158059 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:42.158184 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:42.159236 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0927 15:09:42.159442 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6842,"thread_start_us":6715,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:09:42.160934 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.125167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37197 (local address 127.0.97.2:58623)
0927 15:09:42.131819 (+ 6652us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:42.131890 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:42.134847 (+ 2957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:42.139601 (+ 4754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:42.139615 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:42.140273 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:42.141202 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:42.141232 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.142895 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.142901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:42.145347 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:42.145385 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.146732 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.146740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:42.147158 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.148064 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:42.148101 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:42.151262 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:42.155853 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:42.155866 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:42.155888 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:42.156597 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:42.157129 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:42.157135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:42.157139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:42.157429 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:42.158201 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:42.158211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:42.158667 (+ 456us) client_negotiation.cc:769] Sending connection context
0927 15:09:42.159010 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 15:09:42.159502 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6188,"thread_start_us":7760,"threads_started":1}
I20260927 15:09:42.162608 13105 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37197
I20260927 15:09:42.162968 13105 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:42.163972 13105 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:42.165920 12749 ts_manager.cc:194] Registered new tserver with Master: 57538487bcae4082a93e256ee9733b97 (127.0.97.2:34027)
I20260927 15:09:42.167347 12749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:09:42.192176 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37197
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:42.447026 12962 heartbeater.cc:503] Master 127.0.97.62:37197 was elected leader, sending a full tablet report...
W20260927 15:09:42.546098 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:42.546420 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:42.546514 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:42.554517 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:42.554644 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:42.554757 13111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:09:42.564920 13111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:33055/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:42.566229 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:42.567785 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:42.582520 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:42.582616 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:42.582544 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:42.583264 13111 server_base.cc:1034] running on GCE node
I20260927 15:09:42.583985 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:42.585050 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:42.586299 13111 hybrid_clock.cc:648] HybridClock initialized: now 1790521782586252 us; error 65 us; skew 500 ppm
I20260927 15:09:42.586680 13111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:42.590030 13111 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:09:42.591831 13111 webserver.cc:460] Webserver started at http://127.0.97.3:38053/ using document root <none> and password file <none>
I20260927 15:09:42.592588 13111 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:42.592739 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:42.593065 13111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:42.606787 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "27330ff0685b46bebdcb1d4709066127"
format_stamp: "Formatted at 2026-09-27 15:09:42 on dist-test-slave-g0w1"
server_key: "ab2a33e25c79ff00174d1fdaa156f507"
server_key_iv: "60259c2152828c1276c51202338cc0aa"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:42.607803 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "27330ff0685b46bebdcb1d4709066127"
format_stamp: "Formatted at 2026-09-27 15:09:42 on dist-test-slave-g0w1"
server_key: "ab2a33e25c79ff00174d1fdaa156f507"
server_key_iv: "60259c2152828c1276c51202338cc0aa"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:42.613610 13111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:09:42.629932 13126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:42.632057 13111 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:09:42.632280 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "27330ff0685b46bebdcb1d4709066127"
format_stamp: "Formatted at 2026-09-27 15:09:42 on dist-test-slave-g0w1"
server_key: "ab2a33e25c79ff00174d1fdaa156f507"
server_key_iv: "60259c2152828c1276c51202338cc0aa"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:42.632561 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:42.655460 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:42.659921 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:42.660295 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:42.661306 13111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:42.662827 13111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:42.662900 13111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:42.663123 13111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:42.663188 13111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 15:09:42.697630 13111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43353
I20260927 15:09:42.697721 13238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43353 every 8 connection(s)
I20260927 15:09:42.698992 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:09:42.707511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13111
I20260927 15:09:42.707716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:09:42.708261 388 external_mini_cluster.cc:1428] Setting key 810019c87653d52a3d6735f08b7cdf2d
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:42.720870 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.701736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42893 (local address 127.0.97.62:37197)
0927 15:09:42.702248 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:42.702253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:42.703326 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 15:09:42.705254 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:42.705260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:42.705384 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:42.705511 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:42.707108 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.708004 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.710012 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.710558 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.710709 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.714356 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:42.714361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:42.714365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:42.714405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:42.717474 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.718401 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.718408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.718410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.718565 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.719063 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.719118 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.719121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.719451 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:42.719588 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:42.720419 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 15:09:42.720612 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":233,"threads_started":1}
I20260927 15:09:42.721956 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.702380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37197 (local address 127.0.97.3:42893)
0927 15:09:42.703147 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:42.703189 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:42.704927 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:42.705786 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:42.705796 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:42.706241 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:42.706863 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:42.706881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.708195 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.708200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:42.709856 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:42.709878 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.710805 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.710815 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:42.711175 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.711920 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:42.711944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:42.714080 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:42.717673 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:42.717682 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:42.717699 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:42.718256 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:42.718678 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:42.718681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:42.718683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:42.718931 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:42.719591 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:42.719598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:42.720038 (+ 440us) client_negotiation.cc:769] Sending connection context
0927 15:09:42.720295 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 15:09:42.720728 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":299,"threads_started":1}
I20260927 15:09:42.723852 13239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37197
I20260927 15:09:42.724237 13239 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:42.725100 13239 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:42.726770 12748 ts_manager.cc:194] Registered new tserver with Master: 27330ff0685b46bebdcb1d4709066127 (127.0.97.3:43353)
I20260927 15:09:42.727787 12748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42893
I20260927 15:09:42.737798 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:09:42.739657 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 12829
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:09:42.758245 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.740680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.97.1:43537)
0927 15:09:42.741288 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:42.741300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:42.741384 (+ 84us) server_negotiation.cc:407] Connection header received
0927 15:09:42.741612 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:42.741620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:42.741844 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:42.742018 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:42.743491 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.744667 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.746372 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.746963 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.747139 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.752017 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:42.752033 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:42.752048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:42.752091 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:42.755198 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.755981 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.755997 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.755999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.756150 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.756574 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.756579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.756581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.756939 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:42.757277 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:42.757587 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:09:42.757870 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":274,"threads_started":1}
I20260927 15:09:42.759650 12957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:42.845198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 12829
I20260927 15:09:42.862182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 12969
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:09:42.879755 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.863233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.0.97.2:34027)
0927 15:09:42.863739 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:42.863747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:42.863773 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:09:42.863832 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:42.863838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:42.864054 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:42.864230 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:42.865609 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.866569 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.868183 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.868744 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.868889 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.873476 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:42.873491 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:42.873508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:42.873548 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:42.876608 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.877468 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.877475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.877478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.877624 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.878030 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.878034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.878036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.878427 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:42.878799 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:42.879102 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:09:42.879389 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260927 15:09:42.881166 13100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:42.965375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 12969
I20260927 15:09:42.981534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13111
Sep 27 15:09:42 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:09:43.000449 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:42.982846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.97.3:43353)
0927 15:09:42.983570 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:42.983578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:42.983610 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:09:42.983669 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:42.983676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:42.983921 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:42.984068 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:42.985551 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.986668 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.988267 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:42.988878 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:42.988999 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:42.993900 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:42.993917 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:42.993934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:42.993985 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:42.997223 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.998110 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.998119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.998121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.998269 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:42.998676 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:42.998681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:42.998683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:42.999217 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:42.999483 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:42.999771 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 15:09:43.000021 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":421,"threads_started":1}
I20260927 15:09:43.001900 13234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:43.085845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13111
I20260927 15:09:43.102411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 12717
I20260927 15:09:43.103761 12779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:43.211334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 12717
I20260927 15:09:43.233006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37197
--webserver_interface=127.0.97.62
--webserver_port=42563
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37197
--ranger_config_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:43.577658 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:43.578055 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:43.578220 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:43.585891 13251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:43.586066 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:43.586171 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:43.586273 13251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:43.586356 13251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:43.597113 13251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37197
--ranger_config_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:43.598508 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:43.599994 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:43.612699 13256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:43.612649 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:43.612748 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:43.613651 13251 server_base.cc:1034] running on GCE node
I20260927 15:09:43.614580 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:43.616320 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:43.617596 13251 hybrid_clock.cc:648] HybridClock initialized: now 1790521783617548 us; error 73 us; skew 500 ppm
I20260927 15:09:43.618023 13251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:43 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:43.621497 13251 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:09:43.623274 13251 webserver.cc:460] Webserver started at http://127.0.97.62:42563/ using document root <none> and password file <none>
I20260927 15:09:43.623950 13251 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:43.624092 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:43 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:43.688717 13251 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:09:43.692032 13267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:43.693640 13251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:09:43.693884 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb"
format_stamp: "Formatted at 2026-09-27 15:09:39 on dist-test-slave-g0w1"
server_key: "82b99b24332d77aeb607fc498e1dc0f9"
server_key_iv: "3a69bc65ced30fb4c8241e9763fd919a"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:43.694978 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:43.724448 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:43.728890 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:43.729218 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:43.748757 13251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37197
I20260927 15:09:43.748791 13318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37197 every 8 connection(s)
I20260927 15:09:43.750317 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:43.757491 13319 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:09:43.758929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13251
I20260927 15:09:43.759832 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43537
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37197
--webserver_port=41839
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:43.761039 13319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Bootstrap starting.
I20260927 15:09:43.781153 13319 log.cc:824] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:43.790589 13319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:09:43.791307 13319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Bootstrap complete.
I20260927 15:09:43.800736 13319 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:43.801651 13319 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cec74718e44be7a78c8c4e8a5f8ddb, State: Initialized, Role: FOLLOWER
I20260927 15:09:43.802323 13319 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:43.802572 13319 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:43.802707 13319 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:43.802893 13319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:09:43.806944 13319 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:43.807508 13319 leader_election.cc:302] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0cec74718e44be7a78c8c4e8a5f8ddb; no voters:
I20260927 15:09:43.808189 13319 leader_election.cc:288] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:09:43.808328 13323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:09:43.809702 13323 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [term 2 LEADER]: Becoming Leader. State: Replica: d0cec74718e44be7a78c8c4e8a5f8ddb, State: Running, Role: LEADER
I20260927 15:09:43.810352 13323 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } }
I20260927 15:09:43.810612 13319 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:43.813006 13325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0cec74718e44be7a78c8c4e8a5f8ddb. Latest consensus state: current_term: 2 leader_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } } }
I20260927 15:09:43.813882 13325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:43.813885 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cec74718e44be7a78c8c4e8a5f8ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37197 } } }
I20260927 15:09:43.814275 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:43.824717 13330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:43.832808 13330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:43.834838 13330 catalog_manager.cc:1259] Loaded cluster ID: fc9047d8bfe04431a1089ce5a365a099
I20260927 15:09:43.835026 13330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:43.840440 13330 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:43.842607 13330 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Loaded TSK: 0
I20260927 15:09:43.844718 13330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:09:43.849985 13319 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 15:09:44.236198 13321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:44.236631 13321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:44.236828 13321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:44.248859 13321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:44.249037 13321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:44.249310 13321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:09:44.265664 13321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41839
--tserver_master_addrs=127.0.97.62:37197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:44.267690 13321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:44.269657 13321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:44.319146 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:44.322417 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:44.323637 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:44.324723 13321 server_base.cc:1034] running on GCE node
I20260927 15:09:44.325626 13321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:44.347040 13321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:44.348465 13321 hybrid_clock.cc:648] HybridClock initialized: now 1790521784348348 us; error 113 us; skew 500 ppm
I20260927 15:09:44.348992 13321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:44 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:44.353971 13321 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:09:44.356837 13321 webserver.cc:460] Webserver started at http://127.0.97.1:41839/ using document root <none> and password file <none>
I20260927 15:09:44.357929 13321 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:44.358136 13321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:44 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:44.411415 13321 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.006s sys 0.001s
I20260927 15:09:44.427263 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:44.429522 13321 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.012s
I20260927 15:09:44.429801 13321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "4b7cdba28b67416fb879b6010914cabc"
format_stamp: "Formatted at 2026-09-27 15:09:40 on dist-test-slave-g0w1"
server_key: "821e9b3c0d53a0cc185858524de440a6"
server_key_iv: "ba738b9eb8dcdd17297197cfdbe767b4"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:44.431789 13321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:44.469987 13321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:44.475783 13321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:44.476125 13321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:44.477433 13321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:44.479337 13321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:44.479456 13321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:44.479569 13321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:44.479650 13321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:44.556260 13321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43537
I20260927 15:09:44.556298 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43537 every 8 connection(s)
I20260927 15:09:44.557987 13321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:44.566061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13321
I20260927 15:09:44.566875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34027
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37197
--webserver_port=39039
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:44 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:44.600301 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:44.563564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50271 (local address 127.0.97.62:37197)
0927 15:09:44.569209 (+ 5645us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:44.569232 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:44.569299 (+ 67us) server_negotiation.cc:407] Connection header received
0927 15:09:44.570680 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:44.570705 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:44.571583 (+ 878us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:44.572076 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:44.574822 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:44.576861 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:44.579772 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:44.580994 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:44.581227 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:44.586494 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:44.586511 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:44.586537 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:44.586599 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:44.591628 (+ 5029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:44.593023 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:44.593050 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:44.593061 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:44.593335 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:44.594028 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:44.594050 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:44.594056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:44.594875 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:44.595167 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:44.598356 (+ 3189us) server_negotiation.cc:299] Negotiation successful
0927 15:09:44.598855 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5369,"thread_start_us":5113,"threads_started":1}
I20260927 15:09:44.601250 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:44.562446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37197 (local address 127.0.97.1:50271)
0927 15:09:44.564965 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:44.565018 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:44.569925 (+ 4907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:44.572910 (+ 2985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:44.572924 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:44.573618 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:44.574511 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:44.574541 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:44.577122 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:44.577130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:44.579532 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:44.579563 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:44.581668 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:44.581678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:44.582014 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:44.582914 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:44.582951 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:44.586084 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:44.591919 (+ 5835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:44.591933 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:44.591955 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:44.592782 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:44.593517 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:44.593525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:44.593530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:44.593861 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:44.595285 (+ 1424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:44.595296 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:44.595812 (+ 516us) client_negotiation.cc:769] Sending connection context
0927 15:09:44.596210 (+ 398us) client_negotiation.cc:240] Negotiation successful
0927 15:09:44.596938 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2184,"thread_start_us":1900,"threads_started":1}
I20260927 15:09:44.606420 13481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37197
I20260927 15:09:44.606787 13481 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:44.607776 13481 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
W20260927 15:09:44.612078 13481 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37197 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:09:44.612329 13481 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:44.612643 13481 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:44.614167 13481 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:44.614579 13481 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
W20260927 15:09:44.615936 13481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:09:45.120070 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:45.120637 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:45.120906 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:45.137714 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:45.137998 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:45.138307 13485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:09:45.160210 13485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39039
--tserver_master_addrs=127.0.97.62:37197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:45.162372 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:45.164422 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:09:45.202513 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:45.211162 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:45.227345 13485 server_base.cc:1034] running on GCE node
W20260927 15:09:45.228685 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:45.229707 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:45.232427 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:45.233870 13485 hybrid_clock.cc:648] HybridClock initialized: now 1790521785233785 us; error 75 us; skew 500 ppm
I20260927 15:09:45.234380 13485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:45.244554 13485 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:09:45.246699 13485 webserver.cc:460] Webserver started at http://127.0.97.2:39039/ using document root <none> and password file <none>
I20260927 15:09:45.247689 13485 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:45.247817 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:45.272903 13485 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:09:45.278352 13512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:45.280938 13485 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 15:09:45.281172 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "57538487bcae4082a93e256ee9733b97"
format_stamp: "Formatted at 2026-09-27 15:09:41 on dist-test-slave-g0w1"
server_key: "72a427e2240dcbca79ab5e9829a1014d"
server_key_iv: "b6fac0231d85392e6b751a6efdfac455"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:45.282716 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:45.302376 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:45.308174 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:45.308578 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:45.309897 13485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:45.312006 13485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:45.312227 13485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:09:45.312386 13485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:45.312503 13485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:45.365692 13485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34027
I20260927 15:09:45.365767 13625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34027 every 8 connection(s)
I20260927 15:09:45.367870 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:45.373785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13485
I20260927 15:09:45.374517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33055/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43353
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37197
--webserver_port=38053
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:45.398233 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:45.371421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51363 (local address 127.0.97.62:37197)
0927 15:09:45.371950 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:45.371955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:45.373961 (+ 2006us) server_negotiation.cc:407] Connection header received
0927 15:09:45.376981 (+ 3020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:45.376988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:45.377102 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:45.377213 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:45.379923 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:45.381115 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:45.383930 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:45.384378 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:45.384539 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:45.390289 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:45.390294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:45.390297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:45.390335 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:45.394022 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:45.395171 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:45.395177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:45.395181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:45.395344 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:45.395962 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:45.395968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:45.395972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:45.396362 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:45.396503 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:45.397577 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0927 15:09:45.397827 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
I20260927 15:09:45.400686 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:45.371704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37197 (local address 127.0.97.2:51363)
0927 15:09:45.373736 (+ 2032us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:45.373798 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:45.376524 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:45.377920 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:45.377935 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:45.378629 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:45.379666 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:45.379699 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:45.381386 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:45.381391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:45.383750 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:45.383768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:45.384656 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:45.384665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:45.384940 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:45.386861 (+ 1921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:45.386886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:45.389993 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:45.394245 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:45.394256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:45.394277 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:45.394969 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:45.395512 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:45.395518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:45.395522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:45.395816 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:45.396618 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:45.396627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:45.397074 (+ 447us) client_negotiation.cc:769] Sending connection context
0927 15:09:45.398802 (+ 1728us) client_negotiation.cc:240] Negotiation successful
0927 15:09:45.399447 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1701,"thread_start_us":390,"threads_started":1}
I20260927 15:09:45.402360 13626 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37197
I20260927 15:09:45.402693 13626 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:45.403652 13626 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:45.405764 13284 ts_manager.cc:194] Registered new tserver with Master: 57538487bcae4082a93e256ee9733b97 (127.0.97.2:34027)
I20260927 15:09:45.408085 13284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51363
I20260927 15:09:45.616137 13481 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:45.616652 13481 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:45.618013 13284 ts_manager.cc:194] Registered new tserver with Master: 4b7cdba28b67416fb879b6010914cabc (127.0.97.1:43537)
I20260927 15:09:45.619231 13284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50271
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 15:09:45.859166 13630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:45.859615 13630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:45.859783 13630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:45.867527 13630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:45.867658 13630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:45.867851 13630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:09:45.878333 13630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33055/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38053
--tserver_master_addrs=127.0.97.62:37197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:45.879894 13630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:45.881316 13630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:45.897302 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:45.901950 13638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:45.902081 13635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:45.903908 13630 server_base.cc:1034] running on GCE node
I20260927 15:09:45.904768 13630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:45.907423 13630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:45.908764 13630 hybrid_clock.cc:648] HybridClock initialized: now 1790521785908679 us; error 70 us; skew 500 ppm
I20260927 15:09:45.909294 13630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:45.919579 13630 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:09:45.921627 13630 webserver.cc:460] Webserver started at http://127.0.97.3:38053/ using document root <none> and password file <none>
I20260927 15:09:45.922412 13630 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:45.922598 13630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:45 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:45.961585 13630 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.007s sys 0.003s
I20260927 15:09:45.979825 13646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:45.982060 13630 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.012s
I20260927 15:09:45.982403 13630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "27330ff0685b46bebdcb1d4709066127"
format_stamp: "Formatted at 2026-09-27 15:09:42 on dist-test-slave-g0w1"
server_key: "ab2a33e25c79ff00174d1fdaa156f507"
server_key_iv: "60259c2152828c1276c51202338cc0aa"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:45.984653 13630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:46.036018 13630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:46.042303 13630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:46.042707 13630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:46.050381 13630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:46.058971 13630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:46.059216 13630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:09:46.059399 13630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:46.059509 13630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:09:46 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 15:09:46.198026 13630 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43353
I20260927 15:09:46.198101 13759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43353 every 8 connection(s)
I20260927 15:09:46.200289 13630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:09:46.202960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13630
Sep 27 15:09:46 dist-test-slave-g0w1 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:46.249953 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.204049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.0.97.62:37197)
0927 15:09:46.219550 (+ 15501us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.219557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.219587 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:09:46.219671 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.219677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.219790 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.219922 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:46.222126 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.223278 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.225464 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.226164 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.226297 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.230042 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:46.230049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:46.230054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:46.230118 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:46.237897 (+ 7779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.242953 (+ 5056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.242959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.242963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.243214 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.247397 (+ 4183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.247406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.247411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.247988 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:46.248313 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.249313 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.249556 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15325,"thread_start_us":15470,"threads_started":1}
I20260927 15:09:46.254308 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.206810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45143 (local address 127.0.97.62:37197)
0927 15:09:46.219741 (+ 12931us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.219746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.219760 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:09:46.231233 (+ 11473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.231240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.231340 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.231424 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:46.234335 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.235746 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.239724 (+ 3978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.240440 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.240559 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.245427 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:46.245435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:46.245440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:46.245489 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:46.249326 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.250376 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.250381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.250428 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.250578 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.251784 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.251792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.251798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.252164 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:46.252306 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.253661 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.253921 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12841,"thread_start_us":12459,"threads_started":1}
I20260927 15:09:46.255666 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.206273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37197 (local address 127.0.97.3:45143)
0927 15:09:46.215735 (+ 9462us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:46.215791 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:46.218762 (+ 2971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:46.232210 (+ 13448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:46.232225 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:46.232897 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:46.234051 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.234087 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.236294 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.236299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.238613 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.238632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.240969 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.240977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.241313 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.242177 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:46.242209 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:46.245099 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:46.249518 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:46.249531 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:46.249552 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:46.250231 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:46.250759 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:46.250764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:46.250768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:46.251613 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:46.252427 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:46.252437 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:46.252934 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 15:09:46.253489 (+ 555us) client_negotiation.cc:240] Negotiation successful
0927 15:09:46.254128 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9127,"thread_start_us":8886,"threads_started":1}
I20260927 15:09:46.257670 13760 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37197
I20260927 15:09:46.258037 13760 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:46.258981 13760 heartbeater.cc:511] Master 127.0.97.62:37197 requested a full tablet report, sending...
I20260927 15:09:46.260738 13284 ts_manager.cc:194] Registered new tserver with Master: 27330ff0685b46bebdcb1d4709066127 (127.0.97.3:43353)
I20260927 15:09:46.261941 13284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45143
I20260927 15:09:46.268134 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:09:46.294229 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.275017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.0.97.62:37197)
0927 15:09:46.275254 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.275261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.275705 (+ 444us) server_negotiation.cc:407] Connection header received
0927 15:09:46.275919 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.275925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.276019 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.276154 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:46.278221 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.279429 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.281738 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.282401 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.282523 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.286255 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:46.286263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:46.286268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:46.286317 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:46.289989 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.291176 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.291182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.291186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.291442 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.292062 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.292067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.292071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.292632 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:46.292968 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.293601 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.293868 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:09:46.302487 13284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:09:46.314752 13284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:09:46.376709 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.349441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43353 (local address 127.0.0.1:40778)
0927 15:09:46.356798 (+ 7357us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:46.356856 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:46.357108 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:46.358371 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:46.358383 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:46.358427 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:46.358995 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.359012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.368324 (+ 9312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.368335 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.372195 (+ 3860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.372232 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.375532 (+ 3300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.375538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.375811 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.375889 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 15:09:46.376031 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:09:46.376270 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6992,"thread_start_us":26351,"threads_started":1}
I20260927 15:09:46.377636 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.350000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.97.3:43353)
0927 15:09:46.356011 (+ 6011us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.356025 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.357351 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 15:09:46.357468 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.357476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.357764 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.357964 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:46.359282 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.361476 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.372562 (+ 11086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.374270 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.374461 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.374803 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.377041 (+ 2238us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.377271 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5753,"thread_start_us":5572,"threads_started":1}
I20260927 15:09:46.382093 13694 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9a10f2e5064919aa67e9924597bbcb (DEFAULT_TABLE table=test-table [id=e46b3099fd5c4479b6c58d4d0410f6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:46.384014 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9a10f2e5064919aa67e9924597bbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:46.385906 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.349447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43537 (local address 127.0.0.1:35158)
0927 15:09:46.375523 (+ 26076us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:46.375545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:46.375701 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:46.376662 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:46.376669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:46.376708 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:46.377083 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.377097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.379144 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.379150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.382604 (+ 3454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.382626 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.384997 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.385006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.385199 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.385237 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:09:46.385346 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:09:46.385498 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25916,"mutex_wait_us":1287,"thread_start_us":5500,"threads_started":1}
I20260927 15:09:46.386811 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.349831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.0.97.1:43537)
0927 15:09:46.375368 (+ 25537us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.375375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.375688 (+ 313us) server_negotiation.cc:407] Connection header received
0927 15:09:46.375889 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.375897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.376191 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.376398 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:46.377318 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.378920 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.382841 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.384422 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.384580 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.384813 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.386275 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.386467 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25362,"thread_start_us":25164,"threads_started":1}
I20260927 15:09:46.391611 13416 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9a10f2e5064919aa67e9924597bbcb (DEFAULT_TABLE table=test-table [id=e46b3099fd5c4479b6c58d4d0410f6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:46.395087 13416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9a10f2e5064919aa67e9924597bbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:46.404016 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.357939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34027 (local address 127.0.0.1:56642)
0927 15:09:46.376482 (+ 18543us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:46.376501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:46.376619 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:46.392169 (+ 15550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:46.392176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:46.392214 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:46.392585 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.392601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.398544 (+ 5943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.398552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.400861 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.400889 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.403136 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.403146 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.403338 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.403383 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:09:46.403471 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:09:46.403630 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18417,"thread_start_us":19363,"threads_started":1}
I20260927 15:09:46.405619 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.358205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.97.2:34027)
0927 15:09:46.391019 (+ 32814us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.391029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.391135 (+ 106us) server_negotiation.cc:407] Connection header received
0927 15:09:46.391253 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.391262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.391548 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.391785 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:46.392829 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.394515 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.401192 (+ 6677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.402781 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.404721 (+ 1940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.404998 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.405099 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.405294 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19356,"thread_start_us":21532,"threads_started":1}
I20260927 15:09:46.410388 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9a10f2e5064919aa67e9924597bbcb (DEFAULT_TABLE table=test-table [id=e46b3099fd5c4479b6c58d4d0410f6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:46.412377 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9a10f2e5064919aa67e9924597bbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:46.413663 13626 heartbeater.cc:503] Master 127.0.97.62:37197 was elected leader, sending a full tablet report...
I20260927 15:09:46.423019 13781 tablet_bootstrap.cc:492] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Bootstrap starting.
I20260927 15:09:46.429478 13781 tablet_bootstrap.cc:654] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:46.431033 13781 log.cc:824] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:46.439946 13781 tablet_bootstrap.cc:492] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: No bootstrap required, opened a new log
I20260927 15:09:46.438526 13783 tablet_bootstrap.cc:492] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Bootstrap starting.
I20260927 15:09:46.440402 13781 ts_tablet_manager.cc:1397] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20260927 15:09:46.444265 13783 tablet_bootstrap.cc:654] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:46.445736 13783 log.cc:824] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:46.450244 13788 tablet_bootstrap.cc:492] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Bootstrap starting.
I20260927 15:09:46.451023 13781 raft_consensus.cc:348] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.451768 13781 raft_consensus.cc:374] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:46.451926 13781 raft_consensus.cc:729] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7cdba28b67416fb879b6010914cabc, State: Initialized, Role: FOLLOWER
I20260927 15:09:46.453336 13783 tablet_bootstrap.cc:492] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: No bootstrap required, opened a new log
I20260927 15:09:46.453629 13783 ts_tablet_manager.cc:1397] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260927 15:09:46.454118 13788 tablet_bootstrap.cc:654] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:46.454492 13781 consensus_queue.cc:260] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.455600 13788 log.cc:824] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:46.461112 13781 ts_tablet_manager.cc:1428] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Time spent starting tablet: real 0.020s user 0.011s sys 0.003s
I20260927 15:09:46.462179 13481 heartbeater.cc:503] Master 127.0.97.62:37197 was elected leader, sending a full tablet report...
I20260927 15:09:46.464100 13788 tablet_bootstrap.cc:492] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: No bootstrap required, opened a new log
I20260927 15:09:46.464165 13783 raft_consensus.cc:348] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.464490 13788 ts_tablet_manager.cc:1397] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260927 15:09:46.464640 13783 raft_consensus.cc:374] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:46.464741 13783 raft_consensus.cc:729] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27330ff0685b46bebdcb1d4709066127, State: Initialized, Role: FOLLOWER
I20260927 15:09:46.465451 13783 consensus_queue.cc:260] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.466603 13760 heartbeater.cc:503] Master 127.0.97.62:37197 was elected leader, sending a full tablet report...
I20260927 15:09:46.467234 13783 ts_tablet_manager.cc:1428] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Time spent starting tablet: real 0.013s user 0.015s sys 0.001s
W20260927 15:09:46.470721 13765 tablet.cc:2369] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:09:46.474884 13788 raft_consensus.cc:348] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.475448 13788 raft_consensus.cc:374] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:46.475605 13788 raft_consensus.cc:729] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57538487bcae4082a93e256ee9733b97, State: Initialized, Role: FOLLOWER
I20260927 15:09:46.476341 13788 consensus_queue.cc:260] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.478310 13788 ts_tablet_manager.cc:1428] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
W20260927 15:09:46.569974 13482 tablet.cc:2369] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:09:46.622217 13627 tablet.cc:2369] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:09:46.769413 13793 raft_consensus.cc:482] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:09:46.769809 13793 raft_consensus.cc:504] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.771593 13793 leader_election.cc:288] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7cdba28b67416fb879b6010914cabc (127.0.97.1:43537), 57538487bcae4082a93e256ee9733b97 (127.0.97.2:34027)
I20260927 15:09:46.780174 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.773225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41625 (local address 127.0.97.2:34027)
0927 15:09:46.773414 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.773420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.774092 (+ 672us) server_negotiation.cc:407] Connection header received
0927 15:09:46.774345 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.774350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.774437 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.774543 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:46.775142 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.776059 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.777885 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.779174 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.779316 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.779440 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.779755 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.779909 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:09:46.780296 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.773017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34027 (local address 127.0.97.3:41625)
0927 15:09:46.773948 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:46.773970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:46.774131 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:46.774590 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:46.774596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:46.774621 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:46.774945 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.774957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.776203 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.776211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.777708 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.777723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.779350 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.779357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.779559 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.779596 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:09:46.779708 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:09:46.779858 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":27,"thread_start_us":565,"threads_started":1}
I20260927 15:09:46.780409 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.772104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49417 (local address 127.0.97.1:43537)
0927 15:09:46.772370 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.772379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.772936 (+ 557us) server_negotiation.cc:407] Connection header received
0927 15:09:46.773165 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.773175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.773287 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.773370 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:46.774223 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.775241 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.778021 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.779294 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.779404 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.779493 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.780019 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.780161 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:09:46.781227 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.771883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43537 (local address 127.0.97.3:49417)
0927 15:09:46.772788 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:46.772808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:46.772963 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:46.773648 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:46.773652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:46.773681 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:46.774003 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.774012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.775504 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.775510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.777785 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:46.777800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.779616 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.779621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:46.779823 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.779856 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:09:46.780681 (+ 825us) client_negotiation.cc:240] Negotiation successful
0927 15:09:46.780839 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":5,"thread_start_us":678,"threads_started":1}
I20260927 15:09:46.781849 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9a10f2e5064919aa67e9924597bbcb" candidate_uuid: "27330ff0685b46bebdcb1d4709066127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57538487bcae4082a93e256ee9733b97" is_pre_election: true
I20260927 15:09:46.782430 13581 raft_consensus.cc:2454] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27330ff0685b46bebdcb1d4709066127 in term 0.
I20260927 15:09:46.782399 13436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9a10f2e5064919aa67e9924597bbcb" candidate_uuid: "27330ff0685b46bebdcb1d4709066127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7cdba28b67416fb879b6010914cabc" is_pre_election: true
I20260927 15:09:46.783015 13436 raft_consensus.cc:2454] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27330ff0685b46bebdcb1d4709066127 in term 0.
I20260927 15:09:46.783262 13650 leader_election.cc:302] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27330ff0685b46bebdcb1d4709066127, 57538487bcae4082a93e256ee9733b97; no voters:
I20260927 15:09:46.784014 13793 raft_consensus.cc:2781] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:09:46.784163 13793 raft_consensus.cc:482] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:09:46.784286 13793 raft_consensus.cc:3037] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:46.786749 13793 raft_consensus.cc:504] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.787500 13793 leader_election.cc:288] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [CANDIDATE]: Term 1 election: Requested vote from peers 4b7cdba28b67416fb879b6010914cabc (127.0.97.1:43537), 57538487bcae4082a93e256ee9733b97 (127.0.97.2:34027)
I20260927 15:09:46.788115 13436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9a10f2e5064919aa67e9924597bbcb" candidate_uuid: "27330ff0685b46bebdcb1d4709066127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7cdba28b67416fb879b6010914cabc"
I20260927 15:09:46.788221 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9a10f2e5064919aa67e9924597bbcb" candidate_uuid: "27330ff0685b46bebdcb1d4709066127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57538487bcae4082a93e256ee9733b97"
I20260927 15:09:46.788354 13436 raft_consensus.cc:3037] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:46.788493 13581 raft_consensus.cc:3037] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:46.790907 13436 raft_consensus.cc:2454] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27330ff0685b46bebdcb1d4709066127 in term 1.
I20260927 15:09:46.791599 13650 leader_election.cc:302] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27330ff0685b46bebdcb1d4709066127, 4b7cdba28b67416fb879b6010914cabc; no voters:
I20260927 15:09:46.791931 13793 raft_consensus.cc:2781] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:46.792110 13581 raft_consensus.cc:2454] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27330ff0685b46bebdcb1d4709066127 in term 1.
I20260927 15:09:46.792716 13793 raft_consensus.cc:686] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 1 LEADER]: Becoming Leader. State: Replica: 27330ff0685b46bebdcb1d4709066127, State: Running, Role: LEADER
I20260927 15:09:46.793289 13793 consensus_queue.cc:237] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 15:09:46.798811 13284 catalog_manager.cc:5495] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27330ff0685b46bebdcb1d4709066127 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "27330ff0685b46bebdcb1d4709066127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27330ff0685b46bebdcb1d4709066127" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } health_report { overall_health: UNKNOWN } } }
I20260927 15:09:46.824652 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.818422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.0.97.3:43353)
0927 15:09:46.818646 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.818655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.819000 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:09:46.819293 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.819297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.819386 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.819565 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:09:46.819978 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.820737 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.822166 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.822567 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.822710 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.823344 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:09:46.823900 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:09:46.824073 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.824154 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.824294 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:09:46.835125 13581 raft_consensus.cc:1261] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 1 FOLLOWER]: Refusing update from remote peer 27330ff0685b46bebdcb1d4709066127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:46.835156 13436 raft_consensus.cc:1261] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 1 FOLLOWER]: Refusing update from remote peer 27330ff0685b46bebdcb1d4709066127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:46.836328 13793 consensus_queue.cc:1035] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7cdba28b67416fb879b6010914cabc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:46.837080 13799 consensus_queue.cc:1035] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57538487bcae4082a93e256ee9733b97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:46.856343 13803 mvcc.cc:205] Tried to move back new op lower bound from 7333977238863867904 to 7333977238709587968. Current Snapshot: MvccSnapshot[applied={T|T < 7333977238863867904}]
I20260927 15:09:46.858245 13805 mvcc.cc:205] Tried to move back new op lower bound from 7333977238863867904 to 7333977238709587968. Current Snapshot: MvccSnapshot[applied={T|T < 7333977238863867904}]
I20260927 15:09:46.876936 13283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:46.877321 13283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:46.881770 13283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0cec74718e44be7a78c8c4e8a5f8ddb: Sending DeleteTablet for 3 replicas of tablet 2c9a10f2e5064919aa67e9924597bbcb
I20260927 15:09:46.883239 13416 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c9a10f2e5064919aa67e9924597bbcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35158
I20260927 15:09:46.883246 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c9a10f2e5064919aa67e9924597bbcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40778
I20260927 15:09:46.885110 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c9a10f2e5064919aa67e9924597bbcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56642
I20260927 15:09:46.893986 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13321
I20260927 15:09:46.895948 13812 tablet_replica.cc:330] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: stopping tablet replica
I20260927 15:09:46.896575 13812 raft_consensus.cc:2229] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:46.897158 13812 raft_consensus.cc:2258] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:46.899892 13812 ts_tablet_manager.cc:1905] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:46.907614 13812 ts_tablet_manager.cc:1918] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:46.907881 13812 log.cc:1196] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/2c9a10f2e5064919aa67e9924597bbcb
I20260927 15:09:46.908509 13812 ts_tablet_manager.cc:1939] T 2c9a10f2e5064919aa67e9924597bbcb P 4b7cdba28b67416fb879b6010914cabc: Deleting consensus metadata
I20260927 15:09:46.910588 13271 catalog_manager.cc:4841] TS 4b7cdba28b67416fb879b6010914cabc (127.0.97.1:43537): tablet 2c9a10f2e5064919aa67e9924597bbcb (table test-table [id=e46b3099fd5c4479b6c58d4d0410f6e5]) successfully deleted
I20260927 15:09:46.915132 13814 tablet_replica.cc:330] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: stopping tablet replica
I20260927 15:09:46.915746 13814 raft_consensus.cc:2229] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:46.916390 13814 raft_consensus.cc:2258] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:46.918859 13814 ts_tablet_manager.cc:1905] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:46.919548 13813 tablet_replica.cc:330] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: stopping tablet replica
I20260927 15:09:46.923830 13813 raft_consensus.cc:2229] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:09:46.924685 13813 raft_consensus.cc:2258] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:46.929265 13813 ts_tablet_manager.cc:1905] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:46.934744 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:46.895641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.97.1:43537)
0927 15:09:46.895849 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:46.895855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:46.914384 (+ 18529us) server_negotiation.cc:407] Connection header received
0927 15:09:46.914606 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:46.914613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:46.914712 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:46.914830 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:46.917162 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.918355 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.920549 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:46.921249 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:46.921361 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:46.925504 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:46.925526 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:46.925546 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:46.925601 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:46.930408 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.931618 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.931632 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.931636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.931833 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:46.932384 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:46.932389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:46.932393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:46.932885 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:46.933222 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:46.933873 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 15:09:46.934258 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:09:46.936872 13476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:46.939950 13814 ts_tablet_manager.cc:1918] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:46.940289 13814 log.cc:1196] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/2c9a10f2e5064919aa67e9924597bbcb
I20260927 15:09:46.940968 13814 ts_tablet_manager.cc:1939] T 2c9a10f2e5064919aa67e9924597bbcb P 57538487bcae4082a93e256ee9733b97: Deleting consensus metadata
I20260927 15:09:46.944789 13813 ts_tablet_manager.cc:1918] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:46.946197 13271 catalog_manager.cc:4841] TS 57538487bcae4082a93e256ee9733b97 (127.0.97.2:34027): tablet 2c9a10f2e5064919aa67e9924597bbcb (table test-table [id=e46b3099fd5c4479b6c58d4d0410f6e5]) successfully deleted
I20260927 15:09:46.947248 13813 log.cc:1196] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/2c9a10f2e5064919aa67e9924597bbcb
I20260927 15:09:46.948161 13813 ts_tablet_manager.cc:1939] T 2c9a10f2e5064919aa67e9924597bbcb P 27330ff0685b46bebdcb1d4709066127: Deleting consensus metadata
I20260927 15:09:46.950320 13270 catalog_manager.cc:4841] TS 27330ff0685b46bebdcb1d4709066127 (127.0.97.3:43353): tablet 2c9a10f2e5064919aa67e9924597bbcb (table test-table [id=e46b3099fd5c4479b6c58d4d0410f6e5]) successfully deleted
I20260927 15:09:47.105175 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13321
I20260927 15:09:47.123936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13485
I20260927 15:09:47.139317 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:47.124843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.0.97.2:34027)
0927 15:09:47.125023 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:47.125029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:47.125113 (+ 84us) server_negotiation.cc:407] Connection header received
0927 15:09:47.125341 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:47.125347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:47.125445 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:47.125582 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:47.126898 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:47.127693 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:47.129030 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:47.129485 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:47.129637 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:47.132765 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:47.132781 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:47.132797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:47.132838 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:47.135992 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:47.136863 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:47.136872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:47.136875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:47.137034 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:47.137504 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:47.137510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:47.137514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:47.137925 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:47.138258 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:47.138606 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 15:09:47.138862 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:09:47.140040 13621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:47.244405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13485
I20260927 15:09:47.262935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13630
I20260927 15:09:47.279902 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:47.263968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.97.3:43353)
0927 15:09:47.264192 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:47.264200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:47.264252 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:09:47.264319 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:47.264324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:47.264423 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:47.264557 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:47.265924 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:47.267242 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:47.268534 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:47.269237 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:47.269391 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:47.272061 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:47.272084 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:47.272106 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:47.272160 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:47.276770 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:47.277563 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:47.277573 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:47.277578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:47.277747 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:47.278159 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:47.278165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:47.278168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:47.278539 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:47.278902 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:47.279244 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 15:09:47.279476 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:09:47.280741 13755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:47.407811 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13630
I20260927 15:09:47.427593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13251
I20260927 15:09:47.428848 13314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:47.544400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13251
I20260927 15:09:47.568516 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:09:47.904870 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:09:47.905064 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:09:48.251411 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:09:48.251 UTC [12303] LOG: received smart shutdown request
2026-09-27 15:09:48.255 UTC [12303] LOG: background worker "logical replication launcher" (PID 12312) exited with exit code 1
2026-09-27 15:09:48.255 UTC [12307] LOG: shutting down
2026-09-27 15:09:48.460 UTC [12303] LOG: database system is shut down
2026-09-27T15:09:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57310 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:48 dist-test-slave-g0w1 krb5kdc[13828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:48 dist-test-slave-g0w1 krb5kdc[13828](info): set up 2 sockets
Sep 27 15:09:48 dist-test-slave-g0w1 krb5kdc[13828](info): commencing operation
krb5kdc: starting...
W20260927 15:09:49.699425 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:09:49 dist-test-slave-g0w1 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:09:49.939224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33881
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33881,127.0.97.61:33569,127.0.97.60:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:50.297137 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:50.297541 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:50.297720 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:50.305437 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:50.305627 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:50.305733 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:50.305835 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:50.305914 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:50.316496 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33881,127.0.97.61:33569,127.0.97.60:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:50.317972 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:50.319450 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:50.332413 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.332388 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.332480 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:50.333204 13841 server_base.cc:1034] running on GCE node
I20260927 15:09:50.334103 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:50.335745 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:50.336954 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790521790336893 us; error 61 us; skew 500 ppm
I20260927 15:09:50.337307 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:50 dist-test-slave-g0w1 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:50.340806 13841 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:09:50.342413 13841 webserver.cc:460] Webserver started at http://127.0.97.62:41395/ using document root <none> and password file <none>
I20260927 15:09:50.343281 13841 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:50.343420 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:50.343755 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:50.347018 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "f84d26992f464020a45d20857c9ca1d6"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-g0w1"
server_key: "143cc5968b41d730558b042dc93f1461"
server_key_iv: "1c1e350a9fe79af2115f6edc2fba73ec"
server_key_version: "encryptionkey@0"
I20260927 15:09:50.347970 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "f84d26992f464020a45d20857c9ca1d6"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-g0w1"
server_key: "143cc5968b41d730558b042dc93f1461"
server_key_iv: "1c1e350a9fe79af2115f6edc2fba73ec"
server_key_version: "encryptionkey@0"
I20260927 15:09:50.353914 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:09:50.358031 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.359633 13841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:09:50.359838 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "f84d26992f464020a45d20857c9ca1d6"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-g0w1"
server_key: "143cc5968b41d730558b042dc93f1461"
server_key_iv: "1c1e350a9fe79af2115f6edc2fba73ec"
server_key_version: "encryptionkey@0"
I20260927 15:09:50.360080 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:50.388003 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:50.392722 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:50.393055 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:50.411938 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33881
I20260927 15:09:50.411969 13907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33881 every 8 connection(s)
I20260927 15:09:50.413606 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:50.417598 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:50.423605 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13841
I20260927 15:09:50.423825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:09:50.424463 388 external_mini_cluster.cc:1428] Setting key 3e16efbca16bfd1a7fa12e07e3153e4b
I20260927 15:09:50.424624 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:50 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 15:09:50.448283 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.429265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.0.97.62:33881)
0927 15:09:50.430059 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:50.430070 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:50.430143 (+ 73us) server_negotiation.cc:407] Connection header received
0927 15:09:50.431569 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:50.431575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:50.431825 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:50.432031 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:50.433307 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.434539 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.436056 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.436646 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.436853 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.440372 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:50.440382 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:50.440396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:50.440427 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:50.443413 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.444250 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.444257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.444259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.444407 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.444878 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.444883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.444886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.445587 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:50.445780 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:50.446336 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 15:09:50.446700 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":570,"thread_start_us":392,"threads_started":1}
I20260927 15:09:50.448307 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.427844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33881 (local address 127.0.0.1:43176)
0927 15:09:50.428751 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.428807 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:50.431171 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:50.432045 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:50.432052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:50.432778 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:50.433112 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.433130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.434704 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.434709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.435911 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.435924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.436889 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.436896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.437147 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.437842 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:50.437867 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:50.440138 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:50.443554 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.443567 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.443659 (+ 92us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.444104 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.444514 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.444521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.444527 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.444761 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.445717 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:50.445724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:50.445947 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 15:09:50.446195 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 15:09:50.446815 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":146,"thread_start_us":392,"threads_started":1}
I20260927 15:09:50.451808 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:50.453169 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.452640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51104)
0927 15:09:50.452859 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.453004 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 15:09:50.453469 13859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
I20260927 15:09:50.455199 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.454777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51118)
0927 15:09:50.454925 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.455001 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:09:50.455411 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 15:09:50.474565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33569
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33881,127.0.97.61:33569,127.0.97.60:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:50.504760 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } attempt: 1
I20260927 15:09:50.506019 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.505507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51130)
0927 15:09:50.505744 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.505862 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 15:09:50.507273 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.506796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51144)
0927 15:09:50.506959 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.507038 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 15:09:50.507512 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
I20260927 15:09:50.575795 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } attempt: 2
I20260927 15:09:50.577105 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.576578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51150)
0927 15:09:50.576861 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.576958 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 15:09:50.578342 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.577924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51154)
0927 15:09:50.578078 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.578196 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:09:50.578629 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
I20260927 15:09:50.669924 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } attempt: 3
I20260927 15:09:50.671307 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.670709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51164)
0927 15:09:50.670954 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.671160 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 15:09:50.672422 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.672012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51166)
0927 15:09:50.672213 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.672297 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 15:09:50.672712 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
I20260927 15:09:50.815979 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } attempt: 4
I20260927 15:09:50.817237 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.816651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51172)
0927 15:09:50.816999 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.817089 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 15:09:50.818274 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.817894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51180)
0927 15:09:50.818042 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.818138 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:09:50.818513 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33569: connect: Connection refused (error 111)
W20260927 15:09:50.820326 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:50.820674 13915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:50.820828 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:50.828490 13915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:50.828686 13915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:50.828788 13915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:50.828894 13915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:50.828967 13915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:50.839500 13915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33881,127.0.97.61:33569,127.0.97.60:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:50.840917 13915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:50.842290 13915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:50.854884 13921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.854854 13923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.854810 13920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:50.856020 13915 server_base.cc:1034] running on GCE node
I20260927 15:09:50.856961 13915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:50.858522 13915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:50.859753 13915 hybrid_clock.cc:648] HybridClock initialized: now 1790521790859710 us; error 47 us; skew 500 ppm
I20260927 15:09:50.860132 13915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:50 dist-test-slave-g0w1 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:50.863685 13915 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 15:09:50.865278 13915 webserver.cc:460] Webserver started at http://127.0.97.61:37887/ using document root <none> and password file <none>
I20260927 15:09:50.865979 13915 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:50.866079 13915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:50.866339 13915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:50.869526 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data/instance:
uuid: "1c0d193d82ce44f0be72c9dd8c56dae5"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-g0w1"
server_key: "9cb9b341f7fecdebeb27f3d78bf1064d"
server_key_iv: "ba1f8aab1759271dd6701e70d7edf74d"
server_key_version: "encryptionkey@0"
I20260927 15:09:50.870303 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal/instance:
uuid: "1c0d193d82ce44f0be72c9dd8c56dae5"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-g0w1"
server_key: "9cb9b341f7fecdebeb27f3d78bf1064d"
server_key_iv: "ba1f8aab1759271dd6701e70d7edf74d"
server_key_version: "encryptionkey@0"
I20260927 15:09:50.876099 13915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 15:09:50.880053 13930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.881671 13915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:09:50.881839 13915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal
uuid: "1c0d193d82ce44f0be72c9dd8c56dae5"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-g0w1"
server_key: "9cb9b341f7fecdebeb27f3d78bf1064d"
server_key_iv: "ba1f8aab1759271dd6701e70d7edf74d"
server_key_version: "encryptionkey@0"
I20260927 15:09:50.882033 13915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:50.919255 13915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:50.927613 13915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:50.927901 13915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:50.947472 13915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33569
I20260927 15:09:50.947563 13981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33569 every 8 connection(s)
I20260927 15:09:50.948887 13915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
I20260927 15:09:50.953050 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:50.958567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13915
I20260927 15:09:50.958768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-1/wal/instance
I20260927 15:09:50.959337 388 external_mini_cluster.cc:1428] Setting key b693996bddd4e7c1c10dd9fda1db2c67
I20260927 15:09:50.961040 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:50 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 15:09:50.987684 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.965087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.0.97.62:33881)
0927 15:09:50.965648 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:50.965658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:50.966886 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 15:09:50.969642 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:50.969646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:50.969726 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:50.969837 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:50.972272 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.973049 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.975738 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.976179 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.976298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.981026 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:50.981034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:50.981037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:50.981086 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:50.984032 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.984916 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.984926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.984928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.985099 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.985638 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.985643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.985646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.985933 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:50.986104 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:50.986982 (+ 878us) server_negotiation.cc:299] Negotiation successful
0927 15:09:50.987255 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":260,"threads_started":1}
I20260927 15:09:50.988689 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.965802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33881 (local address 127.0.0.1:43188)
0927 15:09:50.966666 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.966733 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:50.969259 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:50.970133 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:50.970147 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:50.971064 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:50.972020 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.972053 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.973245 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.973254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.975553 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.975574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.976440 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.976449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.976808 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.977690 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:50.977735 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:50.980715 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:50.984236 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.984247 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.984266 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.984761 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.985269 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.985275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.985282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.985502 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.986070 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:50.986078 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:50.986603 (+ 525us) client_negotiation.cc:769] Sending connection context
0927 15:09:50.986875 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 15:09:50.987497 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":375,"threads_started":1}
I20260927 15:09:50.990530 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:50 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:09:51.009621 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.991434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51190)
0927 15:09:50.991635 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.991663 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:50.991782 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:50.992987 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:50.992991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:50.993400 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:50.993563 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.993571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.995802 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.995807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.997008 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.997023 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.998437 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.998442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.998655 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.999418 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:50.999437 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.001539 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.006517 (+ 4978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.006522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.006526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.006976 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.007518 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.007522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.007524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.007690 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.008693 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.008697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.008849 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.008981 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.009278 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":32}
I20260927 15:09:51.010041 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.991583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.0.97.61:33569)
0927 15:09:50.992407 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:50.992418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:50.992455 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:09:50.992518 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:50.992529 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:50.992803 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:50.992989 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:50.993731 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.995602 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.997186 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.998264 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.998421 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.001799 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.001820 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.001844 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.001894 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.006324 (+ 4430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.007171 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.007181 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.007185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.007358 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.007833 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.007838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.007840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.008518 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.008693 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.009256 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.009478 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":618,"thread_start_us":450,"threads_started":1}
I20260927 15:09:51.012907 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:51.014293 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.013761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34810)
0927 15:09:51.014000 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.014121 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 15:09:51.014657 13932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.016511 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.016132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34826)
0927 15:09:51.016296 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.016389 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:09:51.016815 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 15:09:51.018263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38259
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33881,127.0.97.61:33569,127.0.97.60:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:51.066192 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 1
I20260927 15:09:51.067395 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.066895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34836)
0927 15:09:51.067142 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.067233 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:09:51.068681 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.068273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34848)
0927 15:09:51.068475 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.068551 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:09:51.068922 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.117810 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } attempt: 5
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:09:51.134024 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.118814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.0.97.61:33569)
0927 15:09:51.118992 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.118997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.119021 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:09:51.119357 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.119362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.119431 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.119540 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.120845 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.121624 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.123419 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.123913 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.124028 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.128025 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.128030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.128034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.128072 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.131203 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.131994 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.131999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.132002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.132138 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.132561 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.132565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.132567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.132864 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.133024 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.133460 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.133699 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:09:51.134024 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.118685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51204)
0927 15:09:51.118939 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.118963 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.119151 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.119604 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.119609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.120277 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.120609 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.120628 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.121808 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.121813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.123218 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.123236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.124104 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.124110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.124303 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.125103 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.125128 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.127773 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.131370 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.131377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.131386 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.131834 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.132268 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.132278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.132280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.132425 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.132988 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.132993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.133153 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.133332 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.133656 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 15:09:51.135421 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:51.136720 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.136281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34852)
0927 15:09:51.136474 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.136573 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:09:51.137120 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 2
I20260927 15:09:51.137838 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.137454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34858)
0927 15:09:51.137594 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.137698 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:09:51.138095 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.137645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34872)
0927 15:09:51.137824 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.137933 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:09:51.138041 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.139029 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.138720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34882)
0927 15:09:51.138854 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.138920 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:09:51.139336 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.189293 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 1
I20260927 15:09:51.190454 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.189977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34886)
0927 15:09:51.190182 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.190280 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 15:09:51.191525 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.191206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34894)
0927 15:09:51.191350 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.191422 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:09:51.191731 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.230597 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 3
I20260927 15:09:51.231896 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.231391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34910)
0927 15:09:51.231607 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.231717 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:09:51.232988 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.232552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34912)
0927 15:09:51.232793 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.232873 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 15:09:51.233198 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.259938 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 2
I20260927 15:09:51.261119 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.260594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34920)
0927 15:09:51.260828 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.260932 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 15:09:51.262099 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.261776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34930)
0927 15:09:51.261920 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.261992 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:09:51.262333 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
I20260927 15:09:51.368546 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 3
I20260927 15:09:51.369918 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.369280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34934)
0927 15:09:51.369618 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.369725 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 15:09:51.371029 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.370615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34942)
0927 15:09:51.370833 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.370908 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 15:09:51.371322 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
W20260927 15:09:51.372619 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:51.372964 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:51.373111 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:09:51.376427 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 4
I20260927 15:09:51.377650 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.377077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34954)
0927 15:09:51.377336 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.377449 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:09:51.378728 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.378337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34964)
0927 15:09:51.378491 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.378580 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:09:51.379006 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38259: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38259: connect: Connection refused (error 111)
W20260927 15:09:51.381760 13990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:51.381904 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:51.382004 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:51.382090 13990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:51.382154 13990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:51.392844 13990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33881,127.0.97.61:33569,127.0.97.60:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:51.394294 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:51.395738 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:51.408404 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:51.408380 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:51.408354 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:51.409097 13990 server_base.cc:1034] running on GCE node
I20260927 15:09:51.409986 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:51.411751 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:51.412994 13990 hybrid_clock.cc:648] HybridClock initialized: now 1790521791412923 us; error 58 us; skew 500 ppm
I20260927 15:09:51.413426 13990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:51.417487 13990 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 15:09:51.421829 13990 webserver.cc:460] Webserver started at http://127.0.97.60:35595/ using document root <none> and password file <none>
I20260927 15:09:51.422647 13990 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:51.422791 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:51.423208 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:51.426803 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data/instance:
uuid: "0208ddf055134692a3c3db8462a6a59d"
format_stamp: "Formatted at 2026-09-27 15:09:51 on dist-test-slave-g0w1"
server_key: "385fbf91f4674c56b354f95109f77b1d"
server_key_iv: "a259aec8b24cb1fd4acda83d38cf7e06"
server_key_version: "encryptionkey@0"
I20260927 15:09:51.427800 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal/instance:
uuid: "0208ddf055134692a3c3db8462a6a59d"
format_stamp: "Formatted at 2026-09-27 15:09:51 on dist-test-slave-g0w1"
server_key: "385fbf91f4674c56b354f95109f77b1d"
server_key_iv: "a259aec8b24cb1fd4acda83d38cf7e06"
server_key_version: "encryptionkey@0"
I20260927 15:09:51.433696 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:09:51.437500 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.439133 13990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:09:51.439388 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal
uuid: "0208ddf055134692a3c3db8462a6a59d"
format_stamp: "Formatted at 2026-09-27 15:09:51 on dist-test-slave-g0w1"
server_key: "385fbf91f4674c56b354f95109f77b1d"
server_key_iv: "a259aec8b24cb1fd4acda83d38cf7e06"
server_key_version: "encryptionkey@0"
I20260927 15:09:51.439621 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:51.457389 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:51.462007 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:51.462302 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:51.481397 13990 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38259
I20260927 15:09:51.481423 14056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38259 every 8 connection(s)
I20260927 15:09:51.482913 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
I20260927 15:09:51.487210 14057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:51.492202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 13990
I20260927 15:09:51.492419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521577514430-388-0/minicluster-data/master-2/wal/instance
I20260927 15:09:51.492851 388 external_mini_cluster.cc:1428] Setting key 127595bbde4d667c997ed37b23dd5137
I20260927 15:09:51.494710 14057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:09:51.514745 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.498128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43196 (local address 127.0.97.62:33881)
0927 15:09:51.499500 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.499505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.499521 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:09:51.499570 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.499573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.499629 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.499756 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.501579 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.502353 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.503965 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.504388 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.504490 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.508455 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.508461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.508464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.508503 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.511490 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.512275 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.512280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.512282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.512414 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.512879 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.512883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.512885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.513164 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.513335 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.514096 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.514348 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1098,"threads_started":1}
I20260927 15:09:51.515918 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.494914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.97.60:38259)
0927 15:09:51.495836 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.495854 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.495910 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:09:51.498254 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.498286 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.499005 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.499975 (+ 970us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.500879 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.502367 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.503895 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.504469 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.504655 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.508803 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.508814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.508827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.508859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.511847 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.512558 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.512570 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.512574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.512794 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.513175 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.513181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.513184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.513697 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.514008 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.514348 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.514628 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":650,"spinlock_wait_cycles":19456,"thread_start_us":428,"threads_started":1}
I20260927 15:09:51.515954 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.498054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33881 (local address 127.0.0.1:43196)
0927 15:09:51.498770 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.498820 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.498981 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.499823 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.499835 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.500667 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.501342 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.501368 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.502513 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.502521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.503786 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.503798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.504622 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.504628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.504797 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.505761 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.505786 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.508197 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.511655 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.511669 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.511688 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.512123 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.512529 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.512534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.512538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.512750 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.513294 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.513303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.513696 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.513980 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.514507 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":48,"thread_start_us":311,"threads_started":1}
I20260927 15:09:51.518463 14057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:51.533509 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.519348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33569 (local address 127.0.0.1:51210)
0927 15:09:51.519837 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.519855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.519964 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.520284 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.520288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.520684 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.520826 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.520835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.521819 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.521824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.522703 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.522714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.523479 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.523484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.523688 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.524361 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.524385 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.527195 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.530866 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.530872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.530877 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.531371 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.531792 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.531797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.531801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.531955 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.532575 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.532579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.532733 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.532849 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.533150 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362}
I20260927 15:09:51.533540 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.519348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.0.97.61:33569)
0927 15:09:51.519615 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.519620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.519920 (+ 300us) server_negotiation.cc:407] Connection header received
0927 15:09:51.520070 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.520074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.520135 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.520247 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.520989 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.521665 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.522845 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.523328 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.523455 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.527464 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.527474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.527479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.527525 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.530710 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.531494 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.531500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.531513 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.531675 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.532077 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.532081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.532083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.532440 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.532625 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.532996 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.533179 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260927 15:09:51.534757 14057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:51.535337 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.519730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.97.62:33881)
0927 15:09:51.519919 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.519924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.519941 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:09:51.520005 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.520012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.520105 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.520211 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.521447 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.522160 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.523467 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.523912 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.524046 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.529332 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.529341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.529348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.529394 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.532116 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.533000 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.533005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.533009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.533237 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.533712 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.533719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.533723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.534104 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.534354 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.534727 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.534984 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:09:51.548565 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 4
I20260927 15:09:51.550069 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.535889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.97.60:38259)
0927 15:09:51.536039 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.536043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.536057 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:09:51.536126 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.536130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.536182 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.536283 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.537014 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.537974 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.539218 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.539660 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.539761 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.543476 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.543481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.543484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.543514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.546770 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.547622 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.547630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.547662 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.547833 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.548307 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.548311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.548313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.548734 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.548891 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.549374 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.549654 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:09:51.550803 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.535690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34980)
0927 15:09:51.535891 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.535911 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.536020 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.536295 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.536300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.536698 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.536872 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.536886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.538108 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.538111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.539054 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.539067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.539802 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.539810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.539999 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.540604 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.540622 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.543292 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.546925 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.546931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.546937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.547496 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.547982 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.547986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.547989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.548154 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.548945 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.548950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.549121 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.550268 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.550572 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:09:51.555207 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.536990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.0.97.61:33569)
0927 15:09:51.537157 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.537162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.537257 (+ 95us) server_negotiation.cc:407] Connection header received
0927 15:09:51.537447 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.537452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.537518 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.537624 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.538886 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.539773 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.540953 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.541586 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.541728 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.547487 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.547492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.547495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.547532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.552062 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.552877 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.552883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.552887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.553061 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.553523 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.553529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.553533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.553930 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.554213 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.554555 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.554816 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:09:51.557723 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: Bootstrap starting.
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:09:51.561050 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:09:51.561866 14057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:51.563110 14057 log.cc:824] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:51.566550 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: No bootstrap required, opened a new log
I20260927 15:09:51.567319 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.549334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.97.60:38259)
0927 15:09:51.549964 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.549968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.550008 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:09:51.550063 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.550069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.550144 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.551375 (+ 1231us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.551665 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.552973 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.554702 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.555389 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.556081 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.559445 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.559452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.559456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.559500 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.563461 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.564546 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.564552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.564555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.564766 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.565272 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.565276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.565280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.565688 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.565872 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.566458 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.566814 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":352,"threads_started":1}
I20260927 15:09:51.567409 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.549252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:34994)
0927 15:09:51.549451 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.549474 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.549598 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.550356 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.550361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.550897 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.551213 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.551228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.553124 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.553132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.554506 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.554525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.555582 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.555588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.555818 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.556469 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.556496 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.559164 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.563704 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.563712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.563716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.564327 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.564926 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.564931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.564935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.565118 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.565869 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.565878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.566126 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.566342 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.566808 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:09:51.575575 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: Bootstrap starting.
I20260927 15:09:51.577402 14057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.578060 14057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:51.578253 14057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0208ddf055134692a3c3db8462a6a59d, State: Initialized, Role: FOLLOWER
I20260927 15:09:51.579393 14057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.580200 13908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:51.582022 13908 log.cc:824] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:51.587764 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:51.588414 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:51.589196 14057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:51.589988 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: No bootstrap required, opened a new log
I20260927 15:09:51.605518 13908 raft_consensus.cc:348] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.606105 13908 raft_consensus.cc:374] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:51.606256 13908 raft_consensus.cc:729] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84d26992f464020a45d20857c9ca1d6, State: Initialized, Role: FOLLOWER
I20260927 15:09:51.607352 13908 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.608635 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:51.609469 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:51.610543 13908 sys_catalog.cc:564] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:51.620062 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.574405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.0.97.62:33881)
0927 15:09:51.574621 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.574629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.583459 (+ 8830us) server_negotiation.cc:407] Connection header received
0927 15:09:51.583804 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.583810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.583896 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.583982 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.596588 (+ 12606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.597910 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.600086 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.600806 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.600907 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.606984 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.606992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.606997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.607062 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.612566 (+ 5504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.616852 (+ 4286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.616860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.616865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.617105 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.617643 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.617648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.617651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.618095 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.619201 (+ 1106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.619380 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.619636 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:09:51.624226 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.574514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.0.97.61:33569)
0927 15:09:51.575260 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.575267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.596209 (+ 20942us) server_negotiation.cc:407] Connection header received
0927 15:09:51.596304 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.596310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.596400 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.596507 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.599243 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.600330 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.602365 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.602970 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.603124 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.611944 (+ 8820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.611952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.611957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.612009 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.615768 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.621336 (+ 5568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.621346 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.621349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.621553 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.622095 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.622103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.622108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.622531 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.622818 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.623544 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.623829 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607}
I20260927 15:09:51.628451 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.584323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.0.97.60:38259)
0927 15:09:51.584519 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.584524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.604300 (+ 19776us) server_negotiation.cc:407] Connection header received
0927 15:09:51.605390 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.605397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.605518 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.605601 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.609412 (+ 3811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.610643 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.614789 (+ 4146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.615623 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.615780 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.619810 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.619817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.619823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.619870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.624226 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.625074 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.625079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.625083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.625889 (+ 806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.626612 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.626618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.626621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.627003 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.627263 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.627758 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.628085 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 15:09:51.639362 14088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:51.639638 14088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:51.630879 13990 master_runner.cc:186] Error getting master registration for 127.0.97.62:33881: OK, instance_id { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" instance_seqno: 1790521790406653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:09:51.645577 13990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:09:51.646374 13862 tablet.cc:2369] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:09:51.653893 14097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:51.654070 14097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:51.647869 13841 master_runner.cc:186] Error getting master registration for 127.0.97.61:33569: OK, instance_id { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" instance_seqno: 1790521790941836 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:09:51.660032 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:09:51.678392 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } attempt: 5
Sep 27 15:09:51 dist-test-slave-g0w1 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:09:51.697496 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.679419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38259 (local address 127.0.0.1:35020)
0927 15:09:51.679807 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.679829 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.679946 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.680830 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.680835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.681378 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.681597 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.681609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.683185 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.683193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.684702 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.684721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.685811 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.685818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.686028 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.686798 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.686823 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.689910 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.694485 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.694491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.694495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.695098 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.695569 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.695576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.695580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.695769 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.696425 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.696431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.696604 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.696770 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.697094 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260927 15:09:51.697537 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.680165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.97.60:38259)
0927 15:09:51.680379 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.680386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.680405 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:09:51.680469 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.680474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.680546 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.680645 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.681810 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.682988 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.684889 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.685599 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.685736 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.690222 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.690232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.690238 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.690304 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.694296 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.695254 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.695260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.695263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.695436 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.695897 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.695903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.695906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.696293 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.696466 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.696900 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.697143 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:09:51.704149 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: Bootstrap starting.
I20260927 15:09:51.708546 13982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:51.710041 13982 log.cc:824] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: Log is configured to *not* fsync() on all Append() calls
W20260927 15:09:51.714537 14011 tablet.cc:2369] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:09:51.716396 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: No bootstrap required, opened a new log
I20260927 15:09:51.728091 13982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.728574 13982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:51.728690 13982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0d193d82ce44f0be72c9dd8c56dae5, State: Initialized, Role: FOLLOWER
I20260927 15:09:51.729439 13982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.730433 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:51.731148 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:51.731931 13982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:51.752046 13915 master_runner.cc:263] No leader master found from master 1c0d193d82ce44f0be72c9dd8c56dae5
I20260927 15:09:51.752220 13915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:09:51.756461 14111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:51.756620 14111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:51.842428 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:09:51.842763 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.844640 14100 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f84d26992f464020a45d20857c9ca1d6 (127.0.97.62:33881), 0208ddf055134692a3c3db8462a6a59d (127.0.97.60:38259)
I20260927 15:09:51.845955 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84d26992f464020a45d20857c9ca1d6" is_pre_election: true
I20260927 15:09:51.846632 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0d193d82ce44f0be72c9dd8c56dae5 in term 0.
I20260927 15:09:51.847621 14032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0208ddf055134692a3c3db8462a6a59d" is_pre_election: true
I20260927 15:09:51.848254 14032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0d193d82ce44f0be72c9dd8c56dae5 in term 0.
I20260927 15:09:51.849232 13931 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0d193d82ce44f0be72c9dd8c56dae5, f84d26992f464020a45d20857c9ca1d6; no voters:
I20260927 15:09:51.852078 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:09:51.852260 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:09:51.852367 14100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:51.856240 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.857772 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84d26992f464020a45d20857c9ca1d6"
I20260927 15:09:51.858035 13883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:51.861814 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0d193d82ce44f0be72c9dd8c56dae5 in term 1.
I20260927 15:09:51.862548 13931 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0d193d82ce44f0be72c9dd8c56dae5, f84d26992f464020a45d20857c9ca1d6; no voters:
I20260927 15:09:51.866850 14032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0208ddf055134692a3c3db8462a6a59d"
I20260927 15:09:51.867156 14032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:51.872529 14032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0d193d82ce44f0be72c9dd8c56dae5 in term 1.
I20260927 15:09:51.875480 14100 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [CANDIDATE]: Term 1 election: Requested vote from peers f84d26992f464020a45d20857c9ca1d6 (127.0.97.62:33881), 0208ddf055134692a3c3db8462a6a59d (127.0.97.60:38259)
I20260927 15:09:51.876000 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:51.876469 14100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0d193d82ce44f0be72c9dd8c56dae5, State: Running, Role: LEADER
I20260927 15:09:51.877373 14100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } }
I20260927 15:09:51.890642 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0d193d82ce44f0be72c9dd8c56dae5. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:51.891489 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:51.902339 14115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:51.905051 14115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:51.924962 14032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [term 1 FOLLOWER]: Refusing update from remote peer 1c0d193d82ce44f0be72c9dd8c56dae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:51.926275 14113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:09:51.931197 13936 tablet.cc:2369] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:09:51.935693 13883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0d193d82ce44f0be72c9dd8c56dae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:51.937088 14113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:51.973543 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0d193d82ce44f0be72c9dd8c56dae5. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:51.973894 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:51.989254 14116 mvcc.cc:205] Tried to move back new op lower bound from 7333977259710791680 to 7333977259543072768. Current Snapshot: MvccSnapshot[applied={T|T < 7333977259710791680}]
I20260927 15:09:52.019344 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:52.019738 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:52.024715 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:52.025031 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:52.029225 14115 catalog_manager.cc:1347] Generated new cluster ID: 76fb1ee9a0784e01bde45d4a30b01ab5
I20260927 15:09:52.029373 14115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:52.045418 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0d193d82ce44f0be72c9dd8c56dae5. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:52.045786 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:52.054196 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:52.055330 14117 mvcc.cc:205] Tried to move back new op lower bound from 7333977259710791680 to 7333977259543072768. Current Snapshot: MvccSnapshot[applied={T|T < 7333977259710791680}]
I20260927 15:09:52.055171 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:52.067644 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84d26992f464020a45d20857c9ca1d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33881 } } peers { permanent_uuid: "1c0d193d82ce44f0be72c9dd8c56dae5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33569 } } peers { permanent_uuid: "0208ddf055134692a3c3db8462a6a59d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38259 } } }
I20260927 15:09:52.067961 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:52.105975 14115 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:52.108219 14115 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:52.138597 14115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c0d193d82ce44f0be72c9dd8c56dae5: Generated new TSK 0
I20260927 15:09:52.140002 14115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:09:52.641779 14088 catalog_manager.cc:1259] Loaded cluster ID: 76fb1ee9a0784e01bde45d4a30b01ab5
I20260927 15:09:52.641935 14088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: loading cluster ID for follower catalog manager: success
I20260927 15:09:52.644837 14088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: acquiring CA information for follower catalog manager: success
I20260927 15:09:52.646808 14088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0208ddf055134692a3c3db8462a6a59d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:09:52.656082 14097 catalog_manager.cc:1259] Loaded cluster ID: 76fb1ee9a0784e01bde45d4a30b01ab5
I20260927 15:09:52.656240 14097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: loading cluster ID for follower catalog manager: success
I20260927 15:09:52.658799 14097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: acquiring CA information for follower catalog manager: success
I20260927 15:09:52.660753 14097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f84d26992f464020a45d20857c9ca1d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:09:52.742270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13841
I20260927 15:09:52.743708 13903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:52.855589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13841
I20260927 15:09:52.876864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13915
I20260927 15:09:52.878084 13977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:52.988003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13915
I20260927 15:09:53.008749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 13990
I20260927 15:09:53.009927 14052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:53.119978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 13990
2026-09-27T15:09:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4531 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:53 dist-test-slave-g0w1 krb5kdc[14133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:53 dist-test-slave-g0w1 krb5kdc[14133](info): set up 2 sockets
Sep 27 15:09:53 dist-test-slave-g0w1 krb5kdc[14133](info): commencing operation
krb5kdc: starting...
W20260927 15:09:54.247700 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:09:54 dist-test-slave-g0w1 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:09:54.496781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33449
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:54.842317 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:54.842722 14146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:54.842892 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:54.850412 14146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:54.850595 14146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:54.850708 14146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:54.850807 14146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:54.850880 14146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:54.861167 14146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:54.862627 14146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:54.864070 14146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:54.876852 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:54.876828 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:54.876778 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:54.877651 14146 server_base.cc:1034] running on GCE node
I20260927 15:09:54.878556 14146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:54.880312 14146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:54.881597 14146 hybrid_clock.cc:648] HybridClock initialized: now 1790521794881539 us; error 56 us; skew 500 ppm
I20260927 15:09:54.882110 14146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:54 dist-test-slave-g0w1 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:54.885756 14146 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:09:54.887640 14146 webserver.cc:460] Webserver started at http://127.0.97.62:39397/ using document root <none> and password file <none>
I20260927 15:09:54.888438 14146 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:54.888585 14146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:54.888911 14146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:54.892232 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "fdb26798bd2a45998b80bfd6b47120b5"
format_stamp: "Formatted at 2026-09-27 15:09:54 on dist-test-slave-g0w1"
server_key: "fb0263da5c265cf57ba0b4c03f21f4d0"
server_key_iv: "e7e74f8467777bfbdebb96469b1fc4a7"
server_key_version: "encryptionkey@0"
I20260927 15:09:54.893088 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "fdb26798bd2a45998b80bfd6b47120b5"
format_stamp: "Formatted at 2026-09-27 15:09:54 on dist-test-slave-g0w1"
server_key: "fb0263da5c265cf57ba0b4c03f21f4d0"
server_key_iv: "e7e74f8467777bfbdebb96469b1fc4a7"
server_key_version: "encryptionkey@0"
I20260927 15:09:54.898875 14146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:09:54.902649 14161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:54.904143 14146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:54.904379 14146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "fdb26798bd2a45998b80bfd6b47120b5"
format_stamp: "Formatted at 2026-09-27 15:09:54 on dist-test-slave-g0w1"
server_key: "fb0263da5c265cf57ba0b4c03f21f4d0"
server_key_iv: "e7e74f8467777bfbdebb96469b1fc4a7"
server_key_version: "encryptionkey@0"
I20260927 15:09:54.904595 14146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:54.947523 14146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:54.952018 14146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:54.952349 14146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:54.972139 14146 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33449
I20260927 15:09:54.972187 14212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33449 every 8 connection(s)
I20260927 15:09:54.973639 14146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:54.977919 14213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:54.981135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14146
I20260927 15:09:54.981377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:09:54.982228 388 external_mini_cluster.cc:1428] Setting key d12849f0760c76df518a9eea150bdefa
I20260927 15:09:54.991714 14213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5: Bootstrap starting.
I20260927 15:09:54.996004 14213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:54.997561 14213 log.cc:824] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:09:54 dist-test-slave-g0w1 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:55.000986 14213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5: No bootstrap required, opened a new log
I20260927 15:09:55.008244 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.984359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.97.62:33449)
0927 15:09:54.985239 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:54.985251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:54.985298 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:09:54.987197 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:54.987225 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:54.987721 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:54.988108 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:54.990089 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:54.991576 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:54.993531 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:54.994304 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:54.994456 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.000004 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.000013 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.000033 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.000070 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.003529 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.004368 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.004381 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.004388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.004585 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.005015 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.005020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.005028 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.005658 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.005997 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.006504 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.007013 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":426,"threads_started":1}
I20260927 15:09:55.009773 14213 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb26798bd2a45998b80bfd6b47120b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33449 } }
I20260927 15:09:55.010187 14213 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:55.010311 14213 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb26798bd2a45998b80bfd6b47120b5, State: Initialized, Role: FOLLOWER
I20260927 15:09:55.010974 14213 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb26798bd2a45998b80bfd6b47120b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33449 } }
I20260927 15:09:55.011320 14213 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:55.011468 14213 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:55.011665 14213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:55.012632 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:09:55.014185 14213 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb26798bd2a45998b80bfd6b47120b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33449 } }
I20260927 15:09:55.014788 14213 leader_election.cc:302] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb26798bd2a45998b80bfd6b47120b5; no voters:
I20260927 15:09:55.015622 14213 leader_election.cc:288] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:55.015765 14218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:55.016505 14218 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [term 1 LEADER]: Becoming Leader. State: Replica: fdb26798bd2a45998b80bfd6b47120b5, State: Running, Role: LEADER
I20260927 15:09:55.017184 14218 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb26798bd2a45998b80bfd6b47120b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33449 } }
I20260927 15:09:55.018108 14213 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:55.020448 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb26798bd2a45998b80bfd6b47120b5. Latest consensus state: current_term: 1 leader_uuid: "fdb26798bd2a45998b80bfd6b47120b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb26798bd2a45998b80bfd6b47120b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33449 } } }
I20260927 15:09:55.020258 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb26798bd2a45998b80bfd6b47120b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb26798bd2a45998b80bfd6b47120b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33449 } } }
I20260927 15:09:55.020901 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:55.021531 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:55.022398 14224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:55.027742 14224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:55.036885 14224 catalog_manager.cc:1347] Generated new cluster ID: f7a8c84e9e7b4ed7874a98c785a3bb8c
I20260927 15:09:55.037048 14224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:55.064224 14224 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:55.065773 14224 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:55.076654 14224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb26798bd2a45998b80bfd6b47120b5: Generated new TSK 0
I20260927 15:09:55.077584 14224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:09:55.126015 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14146
I20260927 15:09:55.127357 14208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:55.227931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14146
2026-09-27T15:09:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2104 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:55 dist-test-slave-g0w1 krb5kdc[14241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:55 dist-test-slave-g0w1 krb5kdc[14241](info): set up 2 sockets
Sep 27 15:09:55 dist-test-slave-g0w1 krb5kdc[14241](info): commencing operation
krb5kdc: starting...
W20260927 15:09:56.343669 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:09:56 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:09:56.587236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:56.941147 14254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:56.941476 14254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:56.941668 14254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:56.949260 14254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:56.949393 14254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:56.949484 14254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:56.949572 14254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:56.949658 14254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:56.960239 14254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:56.961678 14254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:56.963125 14254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:56.975580 14259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:56.975508 14262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:56.975553 14260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:56.976404 14254 server_base.cc:1034] running on GCE node
I20260927 15:09:56.977289 14254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:56.978881 14254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:56.980137 14254 hybrid_clock.cc:648] HybridClock initialized: now 1790521796980073 us; error 57 us; skew 500 ppm
I20260927 15:09:56.980535 14254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:56 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:56.990219 14254 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:09:56.992177 14254 webserver.cc:460] Webserver started at http://127.0.97.62:34921/ using document root <none> and password file <none>
I20260927 15:09:56.992995 14254 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:56.993153 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:56.993475 14254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:56.996783 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "d1b09daf9c664c648daa03eb63e8d83d"
format_stamp: "Formatted at 2026-09-27 15:09:56 on dist-test-slave-g0w1"
server_key: "e41e6b3cbebef2815304227538fed255"
server_key_iv: "1ab704d793b52605aa014c98fef5a73b"
server_key_version: "encryptionkey@0"
I20260927 15:09:56.997643 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1b09daf9c664c648daa03eb63e8d83d"
format_stamp: "Formatted at 2026-09-27 15:09:56 on dist-test-slave-g0w1"
server_key: "e41e6b3cbebef2815304227538fed255"
server_key_iv: "1ab704d793b52605aa014c98fef5a73b"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.003664 14254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:09:57.007627 14269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:57.009192 14254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:09:57.009403 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d1b09daf9c664c648daa03eb63e8d83d"
format_stamp: "Formatted at 2026-09-27 15:09:56 on dist-test-slave-g0w1"
server_key: "e41e6b3cbebef2815304227538fed255"
server_key_iv: "1ab704d793b52605aa014c98fef5a73b"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.009622 14254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:57.026221 14254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:57.030853 14254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:57.031260 14254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:57.052028 14254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41785
I20260927 15:09:57.052078 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41785 every 8 connection(s)
I20260927 15:09:57.053586 14254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:57.057672 14321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:57.062841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14254
I20260927 15:09:57.063130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:09:57.063959 388 external_mini_cluster.cc:1428] Setting key ce3441169494d8ab792e085f12d4f87f
I20260927 15:09:57.070521 14321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap starting.
I20260927 15:09:57.074662 14321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:57.075853 14321 log.cc:824] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:57.079238 14321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: No bootstrap required, opened a new log
Sep 27 15:09:57 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:57.089995 14321 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:09:57.090499 14321 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:57.090607 14321 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b09daf9c664c648daa03eb63e8d83d, State: Initialized, Role: FOLLOWER
I20260927 15:09:57.091287 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:57.066311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.0.97.62:41785)
0927 15:09:57.067305 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:57.067324 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:57.067391 (+ 67us) server_negotiation.cc:407] Connection header received
0927 15:09:57.069897 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:57.069939 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:57.070536 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:57.071140 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:57.072595 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.074648 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.076319 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.077186 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.077426 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:57.081767 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:57.081779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:57.081800 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:57.081858 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:57.086210 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:57.087065 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:57.087099 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:57.087110 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:57.087343 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:57.087764 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:57.087772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:57.087778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:57.088611 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:57.088935 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:57.089336 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 15:09:57.089830 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":438,"threads_started":1}
I20260927 15:09:57.091373 14321 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:09:57.091670 14321 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:57.091846 14321 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:57.092038 14321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:57.094493 14321 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:09:57.095012 14321 leader_election.cc:302] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b09daf9c664c648daa03eb63e8d83d; no voters:
I20260927 15:09:57.095772 14321 leader_election.cc:288] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:57.095882 14326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:57.098073 14321 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:57.100600 14326 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 LEADER]: Becoming Leader. State: Replica: d1b09daf9c664c648daa03eb63e8d83d, State: Running, Role: LEADER
I20260927 15:09:57.101284 14326 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:09:57.104592 14328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b09daf9c664c648daa03eb63e8d83d. Latest consensus state: current_term: 1 leader_uuid: "d1b09daf9c664c648daa03eb63e8d83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260927 15:09:57.105024 14328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:57.104549 14327 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1b09daf9c664c648daa03eb63e8d83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260927 15:09:57.105425 14327 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: This master's current role is: LEADER
W20260927 15:09:57.109265 14340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:57.109417 14340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:57.109551 14341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:57.111339 14341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:57.119923 14341 catalog_manager.cc:1347] Generated new cluster ID: 0b5ace97456846eaaf6c36e4791fff17
I20260927 15:09:57.120059 14341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:57.153575 14341 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:57.155670 14341 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:57.173718 14341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Generated new TSK 0
I20260927 15:09:57.174901 14341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:09:57.246688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:57.607882 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:57.608311 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:57.608459 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:57.617425 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:57.617566 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:57.617774 14347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:09:57.628341 14347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:57.629750 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:57.631186 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:57.645351 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:57.645306 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:57.645746 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:57.646688 14347 server_base.cc:1034] running on GCE node
I20260927 15:09:57.647480 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:57.648576 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:57.649807 14347 hybrid_clock.cc:648] HybridClock initialized: now 1790521797649742 us; error 67 us; skew 500 ppm
I20260927 15:09:57.650151 14347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:57 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:57.653678 14347 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:09:57.655400 14347 webserver.cc:460] Webserver started at http://127.0.97.1:41923/ using document root <none> and password file <none>
I20260927 15:09:57.656136 14347 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:57.656281 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:57.656590 14347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:57.659716 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "133381142e9940b0bf14ab35785b609f"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-g0w1"
server_key: "a4ad9b46092422f3e2a7700cc00f8498"
server_key_iv: "6dca12643d0d00add8fc2f6c49f9f773"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.660535 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "133381142e9940b0bf14ab35785b609f"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-g0w1"
server_key: "a4ad9b46092422f3e2a7700cc00f8498"
server_key_iv: "6dca12643d0d00add8fc2f6c49f9f773"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.666096 14347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:09:57.669965 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:57.671622 14347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:57.671828 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "133381142e9940b0bf14ab35785b609f"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-g0w1"
server_key: "a4ad9b46092422f3e2a7700cc00f8498"
server_key_iv: "6dca12643d0d00add8fc2f6c49f9f773"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.672068 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:57.700012 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:57.704851 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:57.705166 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:57.706190 14347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:57.707715 14347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:57.707825 14347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:57.707973 14347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:57.708042 14347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:57.742865 14347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41517
I20260927 15:09:57.742910 14474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41517 every 8 connection(s)
I20260927 15:09:57.744377 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:57.754587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14347
I20260927 15:09:57.754843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:09:57.755705 388 external_mini_cluster.cc:1428] Setting key 8e87b16c230e08d9c88d5a26ea25aeb2
Sep 27 15:09:57 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:57.766662 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:57.746966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56925 (local address 127.0.97.62:41785)
0927 15:09:57.747488 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:57.747493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:57.748444 (+ 951us) server_negotiation.cc:407] Connection header received
0927 15:09:57.750509 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:57.750516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:57.750612 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:57.750749 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:57.752560 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.753314 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.755639 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.756130 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.756241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:57.760472 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:57.760477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:57.760480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:57.760513 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:57.763343 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:57.764163 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:57.764168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:57.764171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:57.764335 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:57.764781 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:57.764787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:57.764794 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:57.765130 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:57.765293 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:57.766030 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 15:09:57.766283 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":307,"threads_started":1}
I20260927 15:09:57.767735 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:57.747468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:56925)
0927 15:09:57.748245 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:57.748287 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:57.750168 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:57.751014 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:57.751024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:57.751681 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:57.752349 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:57.752369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.753495 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.753499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:57.755450 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:57.755478 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.756388 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.756394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:57.756653 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:57.757453 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:57.757491 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:57.760224 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:57.763511 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:57.763521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:57.763546 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:57.764016 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:57.764453 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:57.764458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:57.764462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:57.764659 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:57.765257 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:57.765266 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:57.765630 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 15:09:57.765906 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 15:09:57.766500 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":307,"threads_started":1}
I20260927 15:09:57.769486 14475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260927 15:09:57.769845 14475 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:57.770804 14475 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260927 15:09:57.773502 14286 ts_manager.cc:194] Registered new tserver with Master: 133381142e9940b0bf14ab35785b609f (127.0.97.1:41517)
I20260927 15:09:57.775884 14286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56925
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:09:57.811637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:58.162760 14481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:58.163061 14481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:58.163240 14481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:58.171298 14481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:58.171430 14481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:58.171556 14481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:09:58.182078 14481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:58.183549 14481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:58.185006 14481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:58.199703 14489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:58.199761 14487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:58.199792 14486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:58.200419 14481 server_base.cc:1034] running on GCE node
I20260927 15:09:58.201176 14481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:58.202330 14481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:58.203568 14481 hybrid_clock.cc:648] HybridClock initialized: now 1790521798203497 us; error 61 us; skew 500 ppm
I20260927 15:09:58.203924 14481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:58 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:58.207248 14481 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:09:58.208901 14481 webserver.cc:460] Webserver started at http://127.0.97.2:44011/ using document root <none> and password file <none>
I20260927 15:09:58.209635 14481 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:58.209786 14481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:58.210103 14481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:58.213418 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "1ee8e35b8dcd47d197b811e7489d6ba6"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "9f49e6a10ca8470af6f4064a3f9e37ff"
server_key_iv: "b31db017b733eb1b910c08b075372a41"
server_key_version: "encryptionkey@0"
I20260927 15:09:58.214411 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1ee8e35b8dcd47d197b811e7489d6ba6"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "9f49e6a10ca8470af6f4064a3f9e37ff"
server_key_iv: "b31db017b733eb1b910c08b075372a41"
server_key_version: "encryptionkey@0"
I20260927 15:09:58.220300 14481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:09:58.224282 14496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:58.225986 14481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:09:58.226202 14481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "1ee8e35b8dcd47d197b811e7489d6ba6"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "9f49e6a10ca8470af6f4064a3f9e37ff"
server_key_iv: "b31db017b733eb1b910c08b075372a41"
server_key_version: "encryptionkey@0"
I20260927 15:09:58.226430 14481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:58.254791 14481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:58.259440 14481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:58.259707 14481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:58.260679 14481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:58.262055 14481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:58.262130 14481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:58.262215 14481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:58.262259 14481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:58.297066 14481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37177
I20260927 15:09:58.297109 14608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37177 every 8 connection(s)
I20260927 15:09:58.298408 14481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:58.306198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14481
I20260927 15:09:58.306411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:09:58.306835 388 external_mini_cluster.cc:1428] Setting key b563cc8b26826d20dcde2c6015b41dd5
Sep 27 15:09:58 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:58.321877 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.301135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49693 (local address 127.0.97.62:41785)
0927 15:09:58.301652 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:58.301657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:58.302704 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 15:09:58.304883 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:58.304888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:58.304982 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:58.305101 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:58.306846 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.307799 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.309824 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.310614 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.310766 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.314815 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:58.314824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:58.314832 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:58.314890 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:58.318099 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:58.319226 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:58.319231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:58.319233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:58.319389 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:58.319946 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:58.319950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:58.319952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:58.320235 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:58.320360 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:58.321389 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0927 15:09:58.321579 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":309,"threads_started":1}
I20260927 15:09:58.323527 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.301787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:49693)
0927 15:09:58.302536 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:58.302592 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:58.304553 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:58.305395 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:58.305404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:58.305961 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:58.306651 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.306671 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.307998 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.308004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.309672 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.309688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.310841 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.310846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.311054 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.311904 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:58.311927 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:58.314543 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:58.318302 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:58.318317 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:58.318339 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:58.319057 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:58.319511 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:58.319520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:58.319526 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:58.319815 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:58.320371 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:58.320383 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:58.320946 (+ 563us) client_negotiation.cc:769] Sending connection context
0927 15:09:58.321271 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 15:09:58.321907 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":265,"threads_started":1}
I20260927 15:09:58.325206 14609 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260927 15:09:58.325551 14609 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:58.326305 14609 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260927 15:09:58.328006 14286 ts_manager.cc:194] Registered new tserver with Master: 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2:37177)
I20260927 15:09:58.329411 14286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49693
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:09:58.363587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:58.717015 14615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:58.717355 14615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:58.717525 14615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:58.725869 14615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:58.726060 14615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:58.726327 14615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:09:58.738189 14615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:09:58.739693 14615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:58.741204 14615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:58.755833 14623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:58.755856 14621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:58.755921 14620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:58.756935 14615 server_base.cc:1034] running on GCE node
I20260927 15:09:58.757655 14615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:58.758746 14615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:58.759956 14615 hybrid_clock.cc:648] HybridClock initialized: now 1790521798759905 us; error 39 us; skew 500 ppm
I20260927 15:09:58.760304 14615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:09:58 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:58.763708 14615 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:09:58.765426 14615 webserver.cc:460] Webserver started at http://127.0.97.3:40785/ using document root <none> and password file <none>
I20260927 15:09:58.766162 14615 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:58.766317 14615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:58.766634 14615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:58.769822 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "f9cf82a4a1020442357a530dadc557f8"
server_key_iv: "b127fb4c4062b208817d35b85a161ae3"
server_key_version: "encryptionkey@0"
I20260927 15:09:58.770666 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "f9cf82a4a1020442357a530dadc557f8"
server_key_iv: "b127fb4c4062b208817d35b85a161ae3"
server_key_version: "encryptionkey@0"
I20260927 15:09:58.776522 14615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:09:58.780511 14475 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260927 15:09:58.780894 14630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:58.782923 14615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:09:58.783133 14615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "f9cf82a4a1020442357a530dadc557f8"
server_key_iv: "b127fb4c4062b208817d35b85a161ae3"
server_key_version: "encryptionkey@0"
I20260927 15:09:58.783336 14615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:58.812335 14615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:58.817051 14615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:58.817370 14615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:58.818444 14615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:58.819958 14615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:58.820035 14615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:58.820188 14615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:58.820238 14615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:58.855748 14615 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260927 15:09:58.855810 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260927 15:09:58.857164 14615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:09:58.858805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14615
I20260927 15:09:58.858987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:09:58.859639 388 external_mini_cluster.cc:1428] Setting key d3e5a88e8b282e681f50792787ef7dd2
Sep 27 15:09:58 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:09:58.879117 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.860100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46375 (local address 127.0.97.62:41785)
0927 15:09:58.860916 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:58.860921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:58.861872 (+ 951us) server_negotiation.cc:407] Connection header received
0927 15:09:58.864159 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:58.864165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:58.864262 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:58.864378 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:58.866041 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.866861 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.868543 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.868963 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.869103 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.872821 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:58.872826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:58.872830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:58.872873 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:58.876162 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:58.877013 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:58.877017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:58.877019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:58.877138 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:58.877512 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:58.877515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:58.877517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:58.877785 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:58.877894 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:58.878591 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 15:09:58.878790 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":593,"threads_started":1}
I20260927 15:09:58.880117 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.860767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:46375)
0927 15:09:58.861685 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:58.861743 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:58.863883 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:58.864636 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:58.864645 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:58.865197 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:58.865866 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.865887 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.867026 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.867031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.868389 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.868406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.869195 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.869202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.869451 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.870166 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:58.870188 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:58.872555 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:58.876284 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:58.876292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:58.876329 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:58.876908 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:58.877220 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:58.877223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:58.877225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:58.877416 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:58.877885 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:58.877891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:58.878222 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 15:09:58.878495 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 15:09:58.878914 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":350,"threads_started":1}
I20260927 15:09:58.881580 14743 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260927 15:09:58.882009 14743 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:58.882769 14743 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260927 15:09:58.884481 14286 ts_manager.cc:194] Registered new tserver with Master: 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267)
I20260927 15:09:58.885371 14286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46375
I20260927 15:09:58.889084 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:09:58.906658 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.893552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.0.97.62:41785)
0927 15:09:58.893781 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:58.893792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:58.894152 (+ 360us) server_negotiation.cc:407] Connection header received
0927 15:09:58.894317 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:58.894321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:58.894386 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:58.894470 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:58.895952 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.896681 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.898141 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.898589 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.898706 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.901392 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:58.901397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:58.901400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:58.901436 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:58.904212 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:58.904907 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:58.904911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:58.904914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:58.905044 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:58.905373 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:58.905376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:58.905378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:58.905679 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:58.905869 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:58.906140 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:09:58.906336 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:09:58.910879 14286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:09:58.918331 14286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:09:58.942111 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.932291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.0.97.2:37177)
0927 15:09:58.933313 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:58.933321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:58.934023 (+ 702us) server_negotiation.cc:407] Connection header received
0927 15:09:58.934154 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:58.934167 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:58.934699 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:58.934912 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:58.935856 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.937069 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.939870 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.940853 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.941014 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.941240 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:58.941575 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:09:58.941766 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":754,"threads_started":1}
I20260927 15:09:58.942183 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.931647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37177 (local address 127.0.0.1:58752)
0927 15:09:58.933756 (+ 2109us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:58.933775 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:58.933897 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:58.935214 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:58.935221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:58.935259 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:58.935648 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.935661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.937223 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.937233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.939702 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.939721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.941083 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.941090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.941333 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.941374 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:09:58.941519 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 15:09:58.941718 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1975,"mutex_wait_us":1,"thread_start_us":1799,"threads_started":1}
I20260927 15:09:58.943861 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.930950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41517 (local address 127.0.0.1:57886)
0927 15:09:58.933236 (+ 2286us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:58.933297 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:58.933632 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:58.935991 (+ 2359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:58.935998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:58.936028 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:58.936379 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.936391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.938758 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.938763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.941047 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.941065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.943006 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.943013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.943216 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.943252 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:09:58.943322 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:09:58.943456 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"mutex_wait_us":26,"thread_start_us":1850,"threads_started":1}
I20260927 15:09:58.944878 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.933947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.0.1:43216)
0927 15:09:58.934570 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:58.934593 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:58.934748 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:58.937206 (+ 2458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:58.937211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:58.937240 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:58.937673 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.937689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.939663 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.939669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.941866 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:58.941883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.944054 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.944060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:58.944250 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.944283 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:09:58.944377 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:09:58.944506 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":27,"thread_start_us":293,"threads_started":1}
I20260927 15:09:58.945107 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.933968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.0.97.3:44267)
0927 15:09:58.936333 (+ 2365us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:58.936344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:58.936377 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:09:58.936476 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:58.936483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:58.936724 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:58.936916 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:58.937880 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.939511 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.942062 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.943821 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.943971 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.944232 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:58.944526 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 15:09:58.944692 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2161,"thread_start_us":2015,"threads_started":1}
I20260927 15:09:58.945178 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:58.931326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.0.97.1:41517)
0927 15:09:58.932828 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:58.932838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:58.933440 (+ 602us) server_negotiation.cc:407] Connection header received
0927 15:09:58.935463 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:58.935473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:58.935766 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:58.936586 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:58.936882 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.938604 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.941251 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:58.942773 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:58.944222 (+ 1449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:58.944458 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:58.944524 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 15:09:58.944692 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":1068,"threads_started":1}
I20260927 15:09:58.946995 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 233d6f7c2cfa4d85955430a4273836d8 (DEFAULT_TABLE table=test-table [id=f34f1738675c4d1a965813e8390d0e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:58.948575 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233d6f7c2cfa4d85955430a4273836d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:58.948802 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 233d6f7c2cfa4d85955430a4273836d8 (DEFAULT_TABLE table=test-table [id=f34f1738675c4d1a965813e8390d0e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:58.949091 14678 tablet_service.cc:1467] Processing CreateTablet for tablet 233d6f7c2cfa4d85955430a4273836d8 (DEFAULT_TABLE table=test-table [id=f34f1738675c4d1a965813e8390d0e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:58.950222 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233d6f7c2cfa4d85955430a4273836d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:58.950407 14678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233d6f7c2cfa4d85955430a4273836d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:58.963758 14762 tablet_bootstrap.cc:492] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Bootstrap starting.
I20260927 15:09:58.966116 14763 tablet_bootstrap.cc:492] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Bootstrap starting.
I20260927 15:09:58.967983 14762 tablet_bootstrap.cc:654] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:58.968101 14764 tablet_bootstrap.cc:492] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Bootstrap starting.
I20260927 15:09:58.969553 14762 log.cc:824] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:58.970171 14763 tablet_bootstrap.cc:654] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:58.971546 14764 tablet_bootstrap.cc:654] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:58.971597 14763 log.cc:824] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:58.972885 14764 log.cc:824] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:58.973467 14762 tablet_bootstrap.cc:492] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: No bootstrap required, opened a new log
I20260927 15:09:58.973857 14762 ts_tablet_manager.cc:1397] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 15:09:58.974730 14763 tablet_bootstrap.cc:492] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: No bootstrap required, opened a new log
I20260927 15:09:58.974996 14763 ts_tablet_manager.cc:1397] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:09:58.976744 14764 tablet_bootstrap.cc:492] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: No bootstrap required, opened a new log
I20260927 15:09:58.977053 14764 ts_tablet_manager.cc:1397] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:09:58.982987 14763 raft_consensus.cc:348] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:58.983522 14763 raft_consensus.cc:374] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:58.983623 14763 raft_consensus.cc:729] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee8e35b8dcd47d197b811e7489d6ba6, State: Initialized, Role: FOLLOWER
I20260927 15:09:58.984293 14763 consensus_queue.cc:260] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:58.985113 14762 raft_consensus.cc:348] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:58.985450 14609 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260927 15:09:58.985566 14762 raft_consensus.cc:374] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:58.985656 14762 raft_consensus.cc:729] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7177d3bc750c40cb8fc36765bf2e1e71, State: Initialized, Role: FOLLOWER
I20260927 15:09:58.986235 14763 ts_tablet_manager.cc:1428] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260927 15:09:58.986466 14762 consensus_queue.cc:260] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:58.988447 14762 ts_tablet_manager.cc:1428] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260927 15:09:58.988431 14764 raft_consensus.cc:348] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:58.988940 14764 raft_consensus.cc:374] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:58.989042 14764 raft_consensus.cc:729] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133381142e9940b0bf14ab35785b609f, State: Initialized, Role: FOLLOWER
I20260927 15:09:58.989828 14743 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260927 15:09:58.989684 14764 consensus_queue.cc:260] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:58.991600 14764 ts_tablet_manager.cc:1428] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
W20260927 15:09:58.996753 14476 tablet.cc:2369] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:09:59.050763 14610 tablet.cc:2369] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:09:59.109061 14744 tablet.cc:2369] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:09:59.271505 14768 raft_consensus.cc:482] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:09:59.271852 14768 raft_consensus.cc:504] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:59.273540 14768 leader_election.cc:288] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 133381142e9940b0bf14ab35785b609f (127.0.97.1:41517), 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267)
I20260927 15:09:59.280779 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.274065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43559 (local address 127.0.97.1:41517)
0927 15:09:59.274283 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.274289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.274905 (+ 616us) server_negotiation.cc:407] Connection header received
0927 15:09:59.275151 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.275157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.275251 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.275364 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:59.276152 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.276905 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.278885 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.279793 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.279966 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.280078 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.280377 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.280507 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:09:59.280962 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.273985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41517 (local address 127.0.97.2:43559)
0927 15:09:59.274758 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:59.274780 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:59.274930 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:59.275469 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:59.275474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:59.275506 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:59.275974 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.276004 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.277042 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.277051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.278701 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.278719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.280030 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.280036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.280210 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.280235 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:09:59.280348 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:09:59.280493 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"spinlock_wait_cycles":22656,"thread_start_us":431,"threads_started":1}
I20260927 15:09:59.281268 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.275227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56135 (local address 127.0.97.3:44267)
0927 15:09:59.275424 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.275431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.275625 (+ 194us) server_negotiation.cc:407] Connection header received
0927 15:09:59.275835 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.275839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.275896 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.276022 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:59.276616 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.277395 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.279149 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.280253 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.280430 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.280536 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.280853 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.280996 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:09:59.281371 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.274890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.97.2:56135)
0927 15:09:59.275518 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:59.275535 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:59.275703 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:59.276045 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:59.276049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:59.276075 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:59.276463 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.276472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.277541 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.277546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.278972 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.278984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.280487 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.280493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.280667 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.280696 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:09:59.280826 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:09:59.280966 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":21,"spinlock_wait_cycles":1792,"thread_start_us":344,"threads_started":1}
I20260927 15:09:59.282001 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233d6f7c2cfa4d85955430a4273836d8" candidate_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133381142e9940b0bf14ab35785b609f" is_pre_election: true
I20260927 15:09:59.282583 14430 raft_consensus.cc:2454] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee8e35b8dcd47d197b811e7489d6ba6 in term 0.
I20260927 15:09:59.283270 14698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233d6f7c2cfa4d85955430a4273836d8" candidate_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" is_pre_election: true
I20260927 15:09:59.283454 14497 leader_election.cc:302] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133381142e9940b0bf14ab35785b609f, 1ee8e35b8dcd47d197b811e7489d6ba6; no voters:
I20260927 15:09:59.283808 14698 raft_consensus.cc:2454] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee8e35b8dcd47d197b811e7489d6ba6 in term 0.
I20260927 15:09:59.283902 14768 raft_consensus.cc:2781] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:09:59.284078 14768 raft_consensus.cc:482] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:09:59.284204 14768 raft_consensus.cc:3037] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:59.286610 14768 raft_consensus.cc:504] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:59.287326 14768 leader_election.cc:288] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [CANDIDATE]: Term 1 election: Requested vote from peers 133381142e9940b0bf14ab35785b609f (127.0.97.1:41517), 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267)
I20260927 15:09:59.288060 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233d6f7c2cfa4d85955430a4273836d8" candidate_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133381142e9940b0bf14ab35785b609f"
I20260927 15:09:59.288323 14430 raft_consensus.cc:3037] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:59.288493 14698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233d6f7c2cfa4d85955430a4273836d8" candidate_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
I20260927 15:09:59.288705 14698 raft_consensus.cc:3037] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:59.290692 14430 raft_consensus.cc:2454] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee8e35b8dcd47d197b811e7489d6ba6 in term 1.
I20260927 15:09:59.291272 14497 leader_election.cc:302] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133381142e9940b0bf14ab35785b609f, 1ee8e35b8dcd47d197b811e7489d6ba6; no voters:
I20260927 15:09:59.291356 14698 raft_consensus.cc:2454] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee8e35b8dcd47d197b811e7489d6ba6 in term 1.
I20260927 15:09:59.291638 14768 raft_consensus.cc:2781] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:59.292253 14768 raft_consensus.cc:686] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee8e35b8dcd47d197b811e7489d6ba6, State: Running, Role: LEADER
I20260927 15:09:59.292771 14768 consensus_queue.cc:237] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } }
I20260927 15:09:59.298249 14286 catalog_manager.cc:5495] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } health_report { overall_health: HEALTHY } } }
I20260927 15:09:59.402012 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.396064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.0.97.2:37177)
0927 15:09:59.396287 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.396293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.396314 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:09:59.396518 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.396523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.396616 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.396774 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:09:59.397171 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.397930 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.399688 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.400109 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.400259 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.400791 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:09:59.401334 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:09:59.401512 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.401619 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.401712 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:09:59.411504 14698 raft_consensus.cc:1261] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Refusing update from remote peer 1ee8e35b8dcd47d197b811e7489d6ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:59.411504 14430 raft_consensus.cc:1261] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Refusing update from remote peer 1ee8e35b8dcd47d197b811e7489d6ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:59.412686 14773 consensus_queue.cc:1035] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:59.413081 14768 consensus_queue.cc:1035] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:59.443437 14286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:59.443720 14286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:59.447440 14286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Sending DeleteTablet for 3 replicas of tablet 233d6f7c2cfa4d85955430a4273836d8
I20260927 15:09:59.449189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14347
I20260927 15:09:59.449275 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet 233d6f7c2cfa4d85955430a4273836d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58752
I20260927 15:09:59.449275 14678 tablet_service.cc:1514] Processing DeleteTablet for tablet 233d6f7c2cfa4d85955430a4273836d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43216
I20260927 15:09:59.450057 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet 233d6f7c2cfa4d85955430a4273836d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57886
I20260927 15:09:59.460062 14786 tablet_replica.cc:330] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: stopping tablet replica
I20260927 15:09:59.460619 14786 raft_consensus.cc:2229] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:09:59.461440 14786 raft_consensus.cc:2258] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:59.463630 14787 tablet_replica.cc:330] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: stopping tablet replica
I20260927 15:09:59.464145 14787 raft_consensus.cc:2229] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:59.464737 14787 raft_consensus.cc:2258] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:59.464781 14789 tablet_replica.cc:330] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: stopping tablet replica
I20260927 15:09:59.465476 14789 raft_consensus.cc:2229] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:59.466166 14789 raft_consensus.cc:2258] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:59.467547 14787 ts_tablet_manager.cc:1905] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:59.468955 14786 ts_tablet_manager.cc:1905] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:59.470382 14789 ts_tablet_manager.cc:1905] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:59.475760 14787 ts_tablet_manager.cc:1918] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:59.475978 14787 log.cc:1196] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/233d6f7c2cfa4d85955430a4273836d8
I20260927 15:09:59.476549 14787 ts_tablet_manager.cc:1939] T 233d6f7c2cfa4d85955430a4273836d8 P 7177d3bc750c40cb8fc36765bf2e1e71: Deleting consensus metadata
I20260927 15:09:59.477046 14786 ts_tablet_manager.cc:1918] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:59.477270 14786 log.cc:1196] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/233d6f7c2cfa4d85955430a4273836d8
I20260927 15:09:59.477870 14786 ts_tablet_manager.cc:1939] T 233d6f7c2cfa4d85955430a4273836d8 P 1ee8e35b8dcd47d197b811e7489d6ba6: Deleting consensus metadata
I20260927 15:09:59.478803 14789 ts_tablet_manager.cc:1918] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:59.478847 14270 catalog_manager.cc:4841] TS 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267): tablet 233d6f7c2cfa4d85955430a4273836d8 (table test-table [id=f34f1738675c4d1a965813e8390d0e78]) successfully deleted
I20260927 15:09:59.479149 14789 log.cc:1196] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/233d6f7c2cfa4d85955430a4273836d8
I20260927 15:09:59.479877 14789 ts_tablet_manager.cc:1939] T 233d6f7c2cfa4d85955430a4273836d8 P 133381142e9940b0bf14ab35785b609f: Deleting consensus metadata
I20260927 15:09:59.480006 14271 catalog_manager.cc:4841] TS 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2:37177): tablet 233d6f7c2cfa4d85955430a4273836d8 (table test-table [id=f34f1738675c4d1a965813e8390d0e78]) successfully deleted
Sep 27 15:09:59 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:09:59.481667 14270 catalog_manager.cc:4841] TS 133381142e9940b0bf14ab35785b609f (127.0.97.1:41517): tablet 233d6f7c2cfa4d85955430a4273836d8 (table test-table [id=f34f1738675c4d1a965813e8390d0e78]) successfully deleted
I20260927 15:09:59.488855 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.450974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.97.1:41517)
0927 15:09:59.451252 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.451262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.469742 (+ 18480us) server_negotiation.cc:407] Connection header received
0927 15:09:59.470007 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.470013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.470122 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.470232 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:59.472351 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.473516 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.475693 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.476412 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.476541 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.482607 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:59.482619 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:59.482634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:59.482677 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:59.485704 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.486512 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.486520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.486522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.486668 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.487180 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.487185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.487188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.487573 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:59.487972 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.488252 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.488485 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:09:59.489537 14470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:59.623379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14347
I20260927 15:09:59.645882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14481
Sep 27 15:09:59 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:09:59.664651 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.646730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.97.2:37177)
0927 15:09:59.646917 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.646922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.646944 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:09:59.647196 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.647200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.647304 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.647399 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:59.649092 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.649931 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.652003 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.652494 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.652598 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.658037 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:59.658054 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:59.658074 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:59.658127 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:59.661449 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.662271 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.662279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.662281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.662439 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.662894 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.662898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.662900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.663350 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:59.663719 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.664036 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.664267 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:09:59.665329 14604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:59.781138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14481
I20260927 15:09:59.800751 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14615
Sep 27 15:09:59 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:09:59.817303 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.801624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.97.3:44267)
0927 15:09:59.802119 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.802124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.802141 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:09:59.802197 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.802201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.802268 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.802368 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:59.803781 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.804492 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.805736 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.806123 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.806218 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.811011 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:59.811028 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:59.811044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:59.811191 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:59.814166 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.815063 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.815122 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.815128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.815317 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.815711 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.815716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.815718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.816134 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:59.816422 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.816688 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.816891 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
I20260927 15:09:59.817950 14738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:09:59.909191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14615
I20260927 15:09:59.926604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14254
I20260927 15:09:59.927928 14316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:00.047830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14254
I20260927 15:10:00.070286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--webserver_interface=127.0.97.62
--webserver_port=34921
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:00.434249 14795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:00.434582 14795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:00.434741 14795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:00.442657 14795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:00.442816 14795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:00.442926 14795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:00.443042 14795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:00.443137 14795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:00.453799 14795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:00.455368 14795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:00.456832 14795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:00.482710 14801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.483176 14795 server_base.cc:1034] running on GCE node
W20260927 15:10:00.482770 14804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:00.482786 14802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.484552 14795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:00.486641 14795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:00.487903 14795 hybrid_clock.cc:648] HybridClock initialized: now 1790521800487854 us; error 44 us; skew 500 ppm
I20260927 15:10:00.488415 14795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:00 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:00.492841 14795 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:10:00.495158 14795 webserver.cc:460] Webserver started at http://127.0.97.62:34921/ using document root <none> and password file <none>
I20260927 15:10:00.496052 14795 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:00.496340 14795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:00.503720 14795 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 15:10:00.507498 14811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.509310 14795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:10:00.509569 14795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d1b09daf9c664c648daa03eb63e8d83d"
format_stamp: "Formatted at 2026-09-27 15:09:56 on dist-test-slave-g0w1"
server_key: "e41e6b3cbebef2815304227538fed255"
server_key_iv: "1ab704d793b52605aa014c98fef5a73b"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.510811 14795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:00.526572 14795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:00.532243 14795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:00.532610 14795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:00.553925 14795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41785
I20260927 15:10:00.553960 14862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41785 every 8 connection(s)
I20260927 15:10:00.555781 14795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:00.556881 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14795
I20260927 15:10:00.557818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41517
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41785
--webserver_port=41923
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:00.564114 14863 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:10:00.570016 14863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap starting.
I20260927 15:10:00.601742 14863 log.cc:824] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:00.615600 14863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:00.616355 14863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap complete.
I20260927 15:10:00.628317 14863 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:00.629343 14863 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b09daf9c664c648daa03eb63e8d83d, State: Initialized, Role: FOLLOWER
I20260927 15:10:00.630044 14863 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:00.630329 14863 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:00.630465 14863 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:00.630671 14863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:10:00.635270 14863 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:00.635921 14863 leader_election.cc:302] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b09daf9c664c648daa03eb63e8d83d; no voters:
I20260927 15:10:00.636673 14863 leader_election.cc:288] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:10:00.636822 14867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:10:00.638281 14867 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 LEADER]: Becoming Leader. State: Replica: d1b09daf9c664c648daa03eb63e8d83d, State: Running, Role: LEADER
I20260927 15:10:00.639004 14867 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:00.639362 14863 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:00.641680 14869 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b09daf9c664c648daa03eb63e8d83d. Latest consensus state: current_term: 2 leader_uuid: "d1b09daf9c664c648daa03eb63e8d83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260927 15:10:00.642169 14869 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:00.642017 14868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d1b09daf9c664c648daa03eb63e8d83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260927 15:10:00.642642 14868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:00.655045 14875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:00.665177 14875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:00.666911 14875 catalog_manager.cc:1259] Loaded cluster ID: 0b5ace97456846eaaf6c36e4791fff17
I20260927 15:10:00.667092 14875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:10:00.671232 14875 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 15:10:00.693657 14875 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 15:10:00.703737 14875 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:10:00.705687 14875 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Loaded TSK: 0
I20260927 15:10:00.707465 14875 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:10:00.959250 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:00.959640 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:00.959824 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:00.967820 14865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:00.967994 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:00.968192 14865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:10:00.978906 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41923
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:00.980516 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:00.982533 14865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:01.001583 14890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.001555 14891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.002897 14865 server_base.cc:1034] running on GCE node
W20260927 15:10:01.003196 14893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.004029 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:01.005264 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:01.006572 14865 hybrid_clock.cc:648] HybridClock initialized: now 1790521801006501 us; error 52 us; skew 500 ppm
I20260927 15:10:01.007010 14865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:01 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:01.011287 14865 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:10:01.013260 14865 webserver.cc:460] Webserver started at http://127.0.97.1:41923/ using document root <none> and password file <none>
I20260927 15:10:01.014065 14865 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:01.014250 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:01.020979 14865 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:10:01.024680 14900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.026556 14865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:10:01.026811 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "133381142e9940b0bf14ab35785b609f"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-g0w1"
server_key: "a4ad9b46092422f3e2a7700cc00f8498"
server_key_iv: "6dca12643d0d00add8fc2f6c49f9f773"
server_key_version: "encryptionkey@0"
I20260927 15:10:01.028080 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:01.046588 14865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:01.051812 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:01.052142 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:01.053339 14865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:01.054947 14865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:01.055037 14865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.055194 14865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:01.055275 14865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.091890 14865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41517
I20260927 15:10:01.091996 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41517 every 8 connection(s)
I20260927 15:10:01.093446 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:01.100018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 14865
I20260927 15:10:01.100673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37177
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41785
--webserver_port=44011
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:10:01 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:01.121157 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.096668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:39363)
0927 15:10:01.097521 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.097567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.099546 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.101212 (+ 1666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.101226 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.101855 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:01.102440 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.102458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.104360 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.104365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.105608 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.105621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.106706 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.106714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.107009 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.107829 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:01.107852 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:01.110496 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:01.115757 (+ 5261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:01.115767 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:01.115782 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:01.116542 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:01.117194 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:01.117201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:01.117205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:01.117428 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:01.118367 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:01.118376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:01.118768 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.119070 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.119724 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":369,"threads_started":1}
I20260927 15:10:01.121167 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.096501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39363 (local address 127.0.97.62:41785)
0927 15:10:01.097140 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.097151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.097722 (+ 571us) server_negotiation.cc:407] Connection header received
0927 15:10:01.100223 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.100238 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.100606 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.101031 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:01.102683 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.104168 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.105775 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.106448 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.106666 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.110868 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:01.110882 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:01.110907 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:01.110968 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:01.115540 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.116730 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.116743 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.116753 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.117052 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.117565 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.117571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.117575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.118214 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:01.118447 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.119338 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.119751 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":296,"threads_started":1}
I20260927 15:10:01.125020 15013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260927 15:10:01.125458 15013 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:01.126506 15013 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260927 15:10:01.129240 14828 ts_manager.cc:194] Registered new tserver with Master: 133381142e9940b0bf14ab35785b609f (127.0.97.1:41517)
I20260927 15:10:01.131057 14828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39363
W20260927 15:10:01.461751 15017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:01.462057 15017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:01.462203 15017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:01.469691 15017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:01.469822 15017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:01.470018 15017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:10:01.479969 15017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44011
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:01.481367 15017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:01.482750 15017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:01.497012 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.496977 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.496939 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.497967 15017 server_base.cc:1034] running on GCE node
I20260927 15:10:01.498713 15017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:01.499902 15017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:01.501214 15017 hybrid_clock.cc:648] HybridClock initialized: now 1790521801501155 us; error 52 us; skew 500 ppm
I20260927 15:10:01.501595 15017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:01 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:01.505141 15017 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:10:01.507033 15017 webserver.cc:460] Webserver started at http://127.0.97.2:44011/ using document root <none> and password file <none>
I20260927 15:10:01.507874 15017 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:01.507997 15017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:01.513691 15017 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:10:01.516989 15032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.518582 15017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:10:01.518765 15017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "1ee8e35b8dcd47d197b811e7489d6ba6"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "9f49e6a10ca8470af6f4064a3f9e37ff"
server_key_iv: "b31db017b733eb1b910c08b075372a41"
server_key_version: "encryptionkey@0"
I20260927 15:10:01.519989 15017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:01.538291 15017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:01.542585 15017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:01.542830 15017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:01.543812 15017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:01.545082 15017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:01.545181 15017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.545254 15017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:01.545308 15017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.581125 15017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37177
I20260927 15:10:01.581193 15144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37177 every 8 connection(s)
I20260927 15:10:01.582651 15017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:01.587636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15017
I20260927 15:10:01.588239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44267
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41785
--webserver_port=40785
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:10:01 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:01.606122 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.585251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33315 (local address 127.0.97.62:41785)
0927 15:10:01.585477 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.585483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.586714 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 15:10:01.588786 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.588792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.588904 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.589023 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:01.590811 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.591877 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.594339 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.595138 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.595300 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.599600 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:01.599610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:01.599617 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:01.599675 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:01.602664 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.603577 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.603585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.603588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.603728 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.604207 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.604211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.604213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.604497 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:01.604662 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.605510 (+ 848us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.605754 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:10:01.607200 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.585813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:33315)
0927 15:10:01.586538 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.586584 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.588463 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.589295 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.589304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.589842 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:01.590602 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.590628 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.592114 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.592122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.594143 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.594161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.595401 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.595407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.595649 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.596422 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:01.596444 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:01.599232 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:01.602855 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:01.602866 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:01.602885 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:01.603441 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:01.603851 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:01.603857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:01.603861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:01.604073 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:01.604645 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:01.604653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:01.605047 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.605405 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.605978 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":280,"threads_started":1}
I20260927 15:10:01.608808 15145 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260927 15:10:01.609181 15145 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:01.610304 15145 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260927 15:10:01.612155 14828 ts_manager.cc:194] Registered new tserver with Master: 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2:37177)
I20260927 15:10:01.612983 14828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33315
W20260927 15:10:01.936327 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:01.936633 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:01.936793 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:01.944511 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:01.944645 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:01.944830 15148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:10:01.955111 15148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40785
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:01.956938 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:01.958348 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:01.973363 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.973400 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.973306 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.974433 15148 server_base.cc:1034] running on GCE node
I20260927 15:10:01.975199 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:01.976221 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:01.977455 15148 hybrid_clock.cc:648] HybridClock initialized: now 1790521801977411 us; error 41 us; skew 500 ppm
I20260927 15:10:01.977801 15148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:01 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:01.981319 15148 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:10:01.983215 15148 webserver.cc:460] Webserver started at http://127.0.97.3:40785/ using document root <none> and password file <none>
I20260927 15:10:01.983968 15148 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:01.984133 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:01.989964 15148 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:10:01.993160 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.994683 15148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:01.994884 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "f9cf82a4a1020442357a530dadc557f8"
server_key_iv: "b127fb4c4062b208817d35b85a161ae3"
server_key_version: "encryptionkey@0"
I20260927 15:10:01.996025 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:02.028730 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:02.034657 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:02.034981 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:02.036108 15148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:02.037629 15148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:02.037735 15148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.037886 15148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:02.037978 15148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.074065 15148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260927 15:10:02.074134 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260927 15:10:02.075743 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:02.085417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15148
Sep 27 15:10:02 dist-test-slave-g0w1 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:02.102229 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.079358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40297 (local address 127.0.97.62:41785)
0927 15:10:02.079609 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.079614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.081005 (+ 1391us) server_negotiation.cc:407] Connection header received
0927 15:10:02.083275 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.083282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.083401 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.083525 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:02.085666 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.086628 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.088649 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.089096 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.089257 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.094375 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:02.094382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:02.094391 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:02.094440 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:02.098199 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.099309 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.099315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.099319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.099470 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.100057 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.100063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.100066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.100512 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:02.100745 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.101518 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.101894 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:10:02.103472 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.079921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:40297)
0927 15:10:02.080797 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.080847 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:02.082830 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:02.083821 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:02.083831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:02.084516 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:02.085369 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.085406 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.086914 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.086925 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.088458 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.088475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.089367 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.089376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.089646 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.090478 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:02.090515 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:02.094087 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:02.098391 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:02.098404 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:02.098424 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:02.099144 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:02.099598 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:02.099604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:02.099608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:02.099866 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:02.100647 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:02.100658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:02.101021 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 15:10:02.101379 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 15:10:02.101915 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":365,"threads_started":1}
I20260927 15:10:02.103729 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.086472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.0.97.62:41785)
0927 15:10:02.087417 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.087422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.087440 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:10:02.087484 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.087487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.087551 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.087670 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:02.089308 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.090205 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.092394 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.093146 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.093300 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.096837 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:02.096844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:02.096849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:02.096912 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:02.100547 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.101336 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.101343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.101349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.101509 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.102070 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.102075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.102078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.102452 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:02.102719 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.103047 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.103315 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":669,"threads_started":1}
I20260927 15:10:02.104950 15276 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260927 15:10:02.105299 15276 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:02.106053 15276 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260927 15:10:02.107821 14827 ts_manager.cc:194] Registered new tserver with Master: 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267)
I20260927 15:10:02.108861 14827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40297
I20260927 15:10:02.119789 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:02.127610 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.120937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.0.97.62:41785)
0927 15:10:02.121411 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.121418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.121674 (+ 256us) server_negotiation.cc:407] Connection header received
0927 15:10:02.121951 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.121956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.122111 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.122286 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:02.122778 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.123587 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.125130 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.125595 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.125756 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.126328 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:02.126895 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:02.127067 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.127246 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.127343 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260927 15:10:02.128669 14827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:10:02.135691 15013 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
W20260927 15:10:02.136652 14827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:02.161123 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.149981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37177 (local address 127.0.0.1:49976)
0927 15:10:02.150995 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.151050 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:02.151316 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:02.152672 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:02.152694 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:02.152727 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:02.153191 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.153209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.155484 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.155490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.158008 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.158021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.159940 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.159946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.160155 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.160215 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 15:10:02.160360 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 15:10:02.160574 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":17,"thread_start_us":434,"threads_started":1}
I20260927 15:10:02.161710 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.150221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.0.1:47472)
0927 15:10:02.152391 (+ 2170us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.152410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:02.152544 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:02.153632 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:02.153637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:02.153664 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:02.154044 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.154056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.156060 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.156065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.158542 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.158557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.160826 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.160832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.160996 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.161027 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:10:02.161141 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:10:02.161280 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2044,"mutex_wait_us":1,"thread_start_us":1872,"threads_started":1}
I20260927 15:10:02.162132 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.150170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.0.97.2:37177)
0927 15:10:02.151936 (+ 1766us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.151948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.151983 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:10:02.152111 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.152122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.152413 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.153403 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:02.153627 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.155317 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.158201 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.159715 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.159865 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.160075 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.161559 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.161772 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":1485,"threads_started":1}
I20260927 15:10:02.162544 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.150899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.0.97.3:44267)
0927 15:10:02.151754 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.151769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.152677 (+ 908us) server_negotiation.cc:407] Connection header received
0927 15:10:02.152828 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.152842 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.153097 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.153319 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:02.154316 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.155901 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.158727 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.160242 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.160376 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.160647 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.162003 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.162210 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":429,"threads_started":1}
I20260927 15:10:02.162563 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.153417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.0.97.1:41517)
0927 15:10:02.154294 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.154302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.154331 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:10:02.154450 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.154459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.154736 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.154952 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:02.155577 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.157138 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.159433 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.160838 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.161689 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.161972 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.162051 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.162221 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":553,"threads_started":1}
I20260927 15:10:02.163421 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.152780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41517 (local address 127.0.0.1:58306)
0927 15:10:02.153537 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.153556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:02.153704 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:02.154968 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:02.154973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:02.155000 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:02.155404 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.155420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.157322 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.157329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.159273 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.159285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.161135 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.161144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.161679 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.161718 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:10:02.162017 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 15:10:02.162159 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"spinlock_wait_cycles":3456,"thread_start_us":841,"threads_started":1}
I20260927 15:10:02.166435 15211 tablet_service.cc:1467] Processing CreateTablet for tablet 57db302fefc04a2eb557f2d595b88a1b (DEFAULT_TABLE table=test-table [id=59285f7d5c7d46f499195b4b094b7e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:02.166436 15080 tablet_service.cc:1467] Processing CreateTablet for tablet 57db302fefc04a2eb557f2d595b88a1b (DEFAULT_TABLE table=test-table [id=59285f7d5c7d46f499195b4b094b7e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:02.167121 14948 tablet_service.cc:1467] Processing CreateTablet for tablet 57db302fefc04a2eb557f2d595b88a1b (DEFAULT_TABLE table=test-table [id=59285f7d5c7d46f499195b4b094b7e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:02.168020 15211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57db302fefc04a2eb557f2d595b88a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:02.168032 15080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57db302fefc04a2eb557f2d595b88a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:02.168493 14948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57db302fefc04a2eb557f2d595b88a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:02.181625 15292 tablet_bootstrap.cc:492] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Bootstrap starting.
I20260927 15:10:02.186452 15292 tablet_bootstrap.cc:654] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:02.187505 15293 tablet_bootstrap.cc:492] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Bootstrap starting.
I20260927 15:10:02.188519 15294 tablet_bootstrap.cc:492] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Bootstrap starting.
I20260927 15:10:02.188122 15292 log.cc:824] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:02.192538 15293 tablet_bootstrap.cc:654] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:02.192782 15294 tablet_bootstrap.cc:654] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:02.193956 15293 log.cc:824] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:02.194113 15294 log.cc:824] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:02.195992 15292 tablet_bootstrap.cc:492] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: No bootstrap required, opened a new log
I20260927 15:10:02.196416 15292 ts_tablet_manager.cc:1397] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260927 15:10:02.197865 15294 tablet_bootstrap.cc:492] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: No bootstrap required, opened a new log
I20260927 15:10:02.198166 15294 ts_tablet_manager.cc:1397] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 15:10:02.198179 15293 tablet_bootstrap.cc:492] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: No bootstrap required, opened a new log
I20260927 15:10:02.198448 15293 ts_tablet_manager.cc:1397] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260927 15:10:02.206853 15294 raft_consensus.cc:348] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.207405 15294 raft_consensus.cc:374] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:02.207515 15294 raft_consensus.cc:729] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7177d3bc750c40cb8fc36765bf2e1e71, State: Initialized, Role: FOLLOWER
I20260927 15:10:02.207973 15292 raft_consensus.cc:348] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.208233 15294 consensus_queue.cc:260] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.208467 15292 raft_consensus.cc:374] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:02.208586 15292 raft_consensus.cc:729] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133381142e9940b0bf14ab35785b609f, State: Initialized, Role: FOLLOWER
I20260927 15:10:02.209563 15276 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260927 15:10:02.209703 15293 raft_consensus.cc:348] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.210196 15293 raft_consensus.cc:374] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:02.210255 15294 ts_tablet_manager.cc:1428] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 15:10:02.210309 15293 raft_consensus.cc:729] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee8e35b8dcd47d197b811e7489d6ba6, State: Initialized, Role: FOLLOWER
I20260927 15:10:02.210964 15293 consensus_queue.cc:260] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.211047 15292 consensus_queue.cc:260] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.216874 15292 ts_tablet_manager.cc:1428] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Time spent starting tablet: real 0.020s user 0.012s sys 0.006s
I20260927 15:10:02.217025 15293 ts_tablet_manager.cc:1428] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260927 15:10:02.217547 15145 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260927 15:10:02.228713 15299 raft_consensus.cc:482] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:02.229038 15299 raft_consensus.cc:504] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.230824 15299 leader_election.cc:288] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2:37177), 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267)
I20260927 15:10:02.239140 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.231441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48753 (local address 127.0.97.2:37177)
0927 15:10:02.231678 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.231687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.232287 (+ 600us) server_negotiation.cc:407] Connection header received
0927 15:10:02.232531 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.232537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.232610 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.232728 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:02.233831 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.235027 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.237176 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.237998 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.238183 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.238307 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.238648 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.238786 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:10:02.239197 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.231420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48583 (local address 127.0.97.3:44267)
0927 15:10:02.231705 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.231755 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.232339 (+ 584us) server_negotiation.cc:407] Connection header received
0927 15:10:02.232498 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.232504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.232592 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.232728 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:02.233831 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.235027 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.237181 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.238050 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.238225 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.238344 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.238726 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.238883 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:10:02.239364 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.231257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.97.1:48583)
0927 15:10:02.232152 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.232166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:02.232333 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:02.232757 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:02.232762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:02.232811 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:02.233618 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.233631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.235302 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.235308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.236990 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.237006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.238317 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.238326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.238535 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.238560 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:10:02.238672 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:10:02.238864 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":185,"spinlock_wait_cycles":12800,"thread_start_us":541,"threads_started":1}
I20260927 15:10:02.239724 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.231255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37177 (local address 127.0.97.1:48753)
0927 15:10:02.232121 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.232148 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:02.232375 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:02.232766 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:02.232770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:02.232809 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:02.233542 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.233563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.235206 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.235212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.236959 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:02.236978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.238271 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.238280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:02.238466 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.238496 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:10:02.238607 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:10:02.238768 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":135,"spinlock_wait_cycles":73600,"thread_start_us":384,"threads_started":1}
I20260927 15:10:02.240706 15231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57db302fefc04a2eb557f2d595b88a1b" candidate_uuid: "133381142e9940b0bf14ab35785b609f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" is_pre_election: true
I20260927 15:10:02.240774 15100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57db302fefc04a2eb557f2d595b88a1b" candidate_uuid: "133381142e9940b0bf14ab35785b609f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" is_pre_election: true
I20260927 15:10:02.241285 15231 raft_consensus.cc:2454] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133381142e9940b0bf14ab35785b609f in term 0.
I20260927 15:10:02.241284 15100 raft_consensus.cc:2454] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133381142e9940b0bf14ab35785b609f in term 0.
I20260927 15:10:02.242036 14901 leader_election.cc:302] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 133381142e9940b0bf14ab35785b609f, 1ee8e35b8dcd47d197b811e7489d6ba6, 7177d3bc750c40cb8fc36765bf2e1e71; no voters:
I20260927 15:10:02.242465 15299 raft_consensus.cc:2781] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:02.242638 15299 raft_consensus.cc:482] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:02.242776 15299 raft_consensus.cc:3037] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:02.245303 15299 raft_consensus.cc:504] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.246057 15299 leader_election.cc:288] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [CANDIDATE]: Term 1 election: Requested vote from peers 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2:37177), 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267)
I20260927 15:10:02.246805 15231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57db302fefc04a2eb557f2d595b88a1b" candidate_uuid: "133381142e9940b0bf14ab35785b609f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
I20260927 15:10:02.246807 15100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57db302fefc04a2eb557f2d595b88a1b" candidate_uuid: "133381142e9940b0bf14ab35785b609f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6"
I20260927 15:10:02.247102 15100 raft_consensus.cc:3037] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:02.247120 15231 raft_consensus.cc:3037] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:02.250887 15100 raft_consensus.cc:2454] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133381142e9940b0bf14ab35785b609f in term 1.
I20260927 15:10:02.250887 15231 raft_consensus.cc:2454] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133381142e9940b0bf14ab35785b609f in term 1.
I20260927 15:10:02.251533 14901 leader_election.cc:302] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133381142e9940b0bf14ab35785b609f, 7177d3bc750c40cb8fc36765bf2e1e71; no voters:
I20260927 15:10:02.251938 15299 raft_consensus.cc:2781] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:02.252419 15299 raft_consensus.cc:686] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 1 LEADER]: Becoming Leader. State: Replica: 133381142e9940b0bf14ab35785b609f, State: Running, Role: LEADER
I20260927 15:10:02.253052 15299 consensus_queue.cc:237] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } }
I20260927 15:10:02.258368 14827 catalog_manager.cc:5495] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f reported cstate change: term changed from 0 to 1, leader changed from <none> to 133381142e9940b0bf14ab35785b609f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "133381142e9940b0bf14ab35785b609f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133381142e9940b0bf14ab35785b609f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } health_report { overall_health: HEALTHY } } }
I20260927 15:10:02.294179 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.288241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.97.1:41517)
0927 15:10:02.288546 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.288551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.288576 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:10:02.288692 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.288696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.288775 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.288981 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:02.289349 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.290067 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.291792 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.292263 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.292435 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.293006 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:02.293552 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:02.293743 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.293806 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.293896 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 15:10:02.303458 15231 raft_consensus.cc:1261] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Refusing update from remote peer 133381142e9940b0bf14ab35785b609f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:02.303468 15100 raft_consensus.cc:1261] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Refusing update from remote peer 133381142e9940b0bf14ab35785b609f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:02.304575 15303 consensus_queue.cc:1035] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee8e35b8dcd47d197b811e7489d6ba6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:02.304965 15299 consensus_queue.cc:1035] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7177d3bc750c40cb8fc36765bf2e1e71" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:02.317991 15306 mvcc.cc:205] Tried to move back new op lower bound from 7333977302223921152 to 7333977302032908288. Current Snapshot: MvccSnapshot[applied={T|T < 7333977302223921152}]
I20260927 15:10:02.332706 14828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:02.333025 14828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:02.337690 14828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Sending DeleteTablet for 3 replicas of tablet 57db302fefc04a2eb557f2d595b88a1b
I20260927 15:10:02.339179 15080 tablet_service.cc:1514] Processing DeleteTablet for tablet 57db302fefc04a2eb557f2d595b88a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49976
I20260927 15:10:02.339214 14948 tablet_service.cc:1514] Processing DeleteTablet for tablet 57db302fefc04a2eb557f2d595b88a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260927 15:10:02.339381 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14865
I20260927 15:10:02.339577 15211 tablet_service.cc:1514] Processing DeleteTablet for tablet 57db302fefc04a2eb557f2d595b88a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47472
I20260927 15:10:02.351800 15315 tablet_replica.cc:330] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: stopping tablet replica
I20260927 15:10:02.352560 15316 tablet_replica.cc:330] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: stopping tablet replica
I20260927 15:10:02.353207 15316 raft_consensus.cc:2229] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:02.353803 15315 raft_consensus.cc:2229] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:02.354451 15316 raft_consensus.cc:2258] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:02.355237 15315 raft_consensus.cc:2258] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:02.357988 15315 ts_tablet_manager.cc:1905] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:02.363667 15317 tablet_replica.cc:330] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: stopping tablet replica
I20260927 15:10:02.364183 15317 raft_consensus.cc:2229] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:02.364739 15317 raft_consensus.cc:2258] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:02.364996 15316 ts_tablet_manager.cc:1905] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:02.366941 15317 ts_tablet_manager.cc:1905] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:02.367910 15315 ts_tablet_manager.cc:1918] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:02.368181 15315 log.cc:1196] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/57db302fefc04a2eb557f2d595b88a1b
I20260927 15:10:02.368860 15315 ts_tablet_manager.cc:1939] T 57db302fefc04a2eb557f2d595b88a1b P 133381142e9940b0bf14ab35785b609f: Deleting consensus metadata
I20260927 15:10:02.369218 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.340406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.0.97.1:41517)
0927 15:10:02.340583 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.340590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.340982 (+ 392us) server_negotiation.cc:407] Connection header received
0927 15:10:02.341072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.341081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.341188 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.341307 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:02.344244 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.346054 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.350811 (+ 4757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.352045 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.352246 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.358502 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:02.358524 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:02.358548 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:02.358637 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:02.364435 (+ 5798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.365633 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.365700 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.365705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.365907 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.366401 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.366406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.366409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.366926 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:02.367286 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.368553 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.368847 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:10:02.369940 15008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:02.373011 15316 ts_tablet_manager.cc:1918] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:02.373318 15316 log.cc:1196] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/57db302fefc04a2eb557f2d595b88a1b
I20260927 15:10:02.374085 15316 ts_tablet_manager.cc:1939] T 57db302fefc04a2eb557f2d595b88a1b P 1ee8e35b8dcd47d197b811e7489d6ba6: Deleting consensus metadata
I20260927 15:10:02.375469 15317 ts_tablet_manager.cc:1918] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:02.375675 15317 log.cc:1196] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/57db302fefc04a2eb557f2d595b88a1b
I20260927 15:10:02.376302 15317 ts_tablet_manager.cc:1939] T 57db302fefc04a2eb557f2d595b88a1b P 7177d3bc750c40cb8fc36765bf2e1e71: Deleting consensus metadata
I20260927 15:10:02.376382 14813 catalog_manager.cc:4841] TS 1ee8e35b8dcd47d197b811e7489d6ba6 (127.0.97.2:37177): tablet 57db302fefc04a2eb557f2d595b88a1b (table test-table [id=59285f7d5c7d46f499195b4b094b7e2a]) successfully deleted
I20260927 15:10:02.377979 14812 catalog_manager.cc:4841] TS 7177d3bc750c40cb8fc36765bf2e1e71 (127.0.97.3:44267): tablet 57db302fefc04a2eb557f2d595b88a1b (table test-table [id=59285f7d5c7d46f499195b4b094b7e2a]) successfully deleted
I20260927 15:10:02.468724 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14865
I20260927 15:10:02.468962 14812 catalog_manager.cc:4841] TS 133381142e9940b0bf14ab35785b609f (127.0.97.1:41517): tablet 57db302fefc04a2eb557f2d595b88a1b (table test-table [id=59285f7d5c7d46f499195b4b094b7e2a]) successfully deleted
I20260927 15:10:02.488268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15017
I20260927 15:10:02.502751 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.489057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.0.97.2:37177)
0927 15:10:02.489231 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.489235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.489253 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:10:02.489373 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.489378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.489440 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.489526 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:02.491051 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.491995 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.493476 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.493859 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.493953 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.496838 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:02.496855 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:02.496872 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:02.496911 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:02.500038 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.500936 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.500952 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.500955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.501095 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.501443 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.501446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.501448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.501847 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:02.502065 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.502319 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.502493 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:10:02.503561 15140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:02.596609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15017
I20260927 15:10:02.615017 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15148
I20260927 15:10:02.630939 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.616006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.0.97.3:44267)
0927 15:10:02.616254 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:02.616260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:02.616284 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:10:02.616353 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:02.616357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:02.616444 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:02.616537 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:02.617987 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.618756 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.620229 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:02.620717 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:02.620841 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:02.623660 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:02.623678 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:02.623695 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:02.623739 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:02.627207 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.628125 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.628138 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.628144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.628350 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:02.628970 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:02.628975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:02.628977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:02.629522 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:02.629883 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:02.630304 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:10:02.630571 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:10:02.631795 15271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:02.724012 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15148
I20260927 15:10:02.740907 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 14795
I20260927 15:10:02.742025 14858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:02.858883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 14795
I20260927 15:10:02.881093 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--webserver_interface=127.0.97.62
--webserver_port=34921
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:03.232699 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:03.233018 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:03.233184 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:03.241101 15322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:03.241228 15322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:03.241307 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:03.241415 15322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:03.241492 15322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:03.251942 15322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:03.253512 15322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:03.254963 15322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:03.268133 15327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:03.268004 15328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:03.267976 15330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:03.268778 15322 server_base.cc:1034] running on GCE node
I20260927 15:10:03.269667 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:03.271299 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:03.272547 15322 hybrid_clock.cc:648] HybridClock initialized: now 1790521803272488 us; error 52 us; skew 500 ppm
I20260927 15:10:03.272936 15322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:03 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:03.276403 15322 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:10:03.278106 15322 webserver.cc:460] Webserver started at http://127.0.97.62:34921/ using document root <none> and password file <none>
I20260927 15:10:03.278891 15322 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:03.279053 15322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:03.284964 15322 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 15:10:03.288558 15337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:03.290534 15322 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:10:03.290815 15322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d1b09daf9c664c648daa03eb63e8d83d"
format_stamp: "Formatted at 2026-09-27 15:09:56 on dist-test-slave-g0w1"
server_key: "e41e6b3cbebef2815304227538fed255"
server_key_iv: "1ab704d793b52605aa014c98fef5a73b"
server_key_version: "encryptionkey@0"
I20260927 15:10:03.292053 15322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:03.305004 15322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:03.309512 15322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:03.309841 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:03.330430 15322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41785
I20260927 15:10:03.330447 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41785 every 8 connection(s)
I20260927 15:10:03.332290 15322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:03.335737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15322
I20260927 15:10:03.336781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41517
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41785
--webserver_port=41923
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:03.342041 15389 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:10:03.347046 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap starting.
I20260927 15:10:03.369642 15389 log.cc:824] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:03.387718 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:03.388298 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d: Bootstrap complete.
I20260927 15:10:03.398412 15389 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:03.399437 15389 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b09daf9c664c648daa03eb63e8d83d, State: Initialized, Role: FOLLOWER
I20260927 15:10:03.400126 15389 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:03.400380 15389 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:03.400512 15389 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:03.400692 15389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:10:03.404795 15389 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:03.405332 15389 leader_election.cc:302] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b09daf9c664c648daa03eb63e8d83d; no voters:
I20260927 15:10:03.406090 15389 leader_election.cc:288] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:10:03.406255 15393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:10:03.407490 15393 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [term 3 LEADER]: Becoming Leader. State: Replica: d1b09daf9c664c648daa03eb63e8d83d, State: Running, Role: LEADER
I20260927 15:10:03.408073 15393 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260927 15:10:03.408460 15389 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:03.410840 15395 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b09daf9c664c648daa03eb63e8d83d. Latest consensus state: current_term: 3 leader_uuid: "d1b09daf9c664c648daa03eb63e8d83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260927 15:10:03.410970 15394 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d1b09daf9c664c648daa03eb63e8d83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b09daf9c664c648daa03eb63e8d83d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260927 15:10:03.411319 15395 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:03.411427 15394 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b09daf9c664c648daa03eb63e8d83d [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:03.420909 15400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:03.430981 15400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:03.432777 15400 catalog_manager.cc:1259] Loaded cluster ID: 0b5ace97456846eaaf6c36e4791fff17
I20260927 15:10:03.432912 15400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 15:10:03.435153 15400 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f12d40929ad google::LogMessage::Fail() at ??:0
@ 0x7f12d409588a google::LogMessage::SendToLog() at ??:0
@ 0x7f12d40924de google::LogMessage::Flush() at ??:0
@ 0x7f12d4093a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12e9ac9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f12e9ac6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f12e9b4e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f12e9b4e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f12d6cde345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f12d6ce191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f12d6ce1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f12d6cbceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12d9f966db start_thread at ??:0
@ 0x7f12d2f2c61f clone at ??:0
W20260927 15:10:03.756934 15391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:03.757249 15391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:03.757357 15391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:03.765048 15391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:03.765183 15391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:03.765429 15391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:10:03.775665 15391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41923
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:03.776994 15391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:03.778324 15391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:03.791929 15430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:03.791982 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:03.792007 15431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:03.792936 15391 server_base.cc:1034] running on GCE node
I20260927 15:10:03.793612 15391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:03.794677 15391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:03.795946 15391 hybrid_clock.cc:648] HybridClock initialized: now 1790521803795882 us; error 58 us; skew 500 ppm
I20260927 15:10:03.796365 15391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:03 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:03.799865 15391 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:10:03.801527 15391 webserver.cc:460] Webserver started at http://127.0.97.1:41923/ using document root <none> and password file <none>
I20260927 15:10:03.802261 15391 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:03.802418 15391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:03.808171 15391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:10:03.811244 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:03.812748 15391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:10:03.812953 15391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "133381142e9940b0bf14ab35785b609f"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-g0w1"
server_key: "a4ad9b46092422f3e2a7700cc00f8498"
server_key_iv: "6dca12643d0d00add8fc2f6c49f9f773"
server_key_version: "encryptionkey@0"
I20260927 15:10:03.814025 15391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:03.830127 15391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:03.834720 15391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:03.835041 15391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:03.836202 15391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:03.837644 15391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:03.837783 15391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:03.837920 15391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:03.838017 15391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:03.873919 15391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41517
I20260927 15:10:03.874017 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41517 every 8 connection(s)
I20260927 15:10:03.875460 15391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:03.880756 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.878641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:57439)
0927 15:10:03.879526 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.879750 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":417,"threads_started":1}
W20260927 15:10:03.881497 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41785 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41785: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
I20260927 15:10:03.882489 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.882057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:46357)
0927 15:10:03.882270 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.882347 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 15:10:03.883634 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.883254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:49929)
0927 15:10:03.883414 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.883509 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:10:03.883919 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:10:03.885617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15391
I20260927 15:10:03.886261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37177
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41785
--webserver_port=44011
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:04.247236 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:04.247566 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:04.247663 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:04.255362 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:04.255497 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:04.255613 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:10:04.265899 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44011
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:04.267414 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:04.268792 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:04.283322 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:04.283247 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:04.283329 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:04.284231 15556 server_base.cc:1034] running on GCE node
I20260927 15:10:04.284922 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:04.286023 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:04.287231 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790521804287171 us; error 42 us; skew 500 ppm
I20260927 15:10:04.287596 15556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:04 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:04.291195 15556 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:10:04.292884 15556 webserver.cc:460] Webserver started at http://127.0.97.2:44011/ using document root <none> and password file <none>
I20260927 15:10:04.293642 15556 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:04.293790 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:04.299681 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:10:04.302927 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:04.304662 15556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:04.304935 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "1ee8e35b8dcd47d197b811e7489d6ba6"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "9f49e6a10ca8470af6f4064a3f9e37ff"
server_key_iv: "b31db017b733eb1b910c08b075372a41"
server_key_version: "encryptionkey@0"
I20260927 15:10:04.306121 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:04.328193 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:04.332787 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:04.333094 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:04.334173 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:04.335789 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:04.335883 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:04.336058 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:04.336146 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:04.372008 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37177
I20260927 15:10:04.372067 15683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37177 every 8 connection(s)
I20260927 15:10:04.373653 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:04.379695 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.377364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:58851)
0927 15:10:04.378301 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.378557 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":370,"threads_started":1}
W20260927 15:10:04.380470 15684 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41785 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41785: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
I20260927 15:10:04.381594 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.381206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:50451)
0927 15:10:04.381404 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.381478 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 15:10:04.381562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15556
I20260927 15:10:04.382256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44267
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41785
--webserver_port=40785
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:04.382540 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.382142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:56679)
0927 15:10:04.382294 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.382372 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:10:04.382797 15684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:10:04.736500 15687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:04.736811 15687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:04.736986 15687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:04.744529 15687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:04.744650 15687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:04.744860 15687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:10:04.755692 15687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40785
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:04.757157 15687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:04.758522 15687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:04.772629 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:04.772704 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:04.772641 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:04.773792 15687 server_base.cc:1034] running on GCE node
I20260927 15:10:04.774503 15687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:04.775561 15687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:04.776819 15687 hybrid_clock.cc:648] HybridClock initialized: now 1790521804776768 us; error 47 us; skew 500 ppm
I20260927 15:10:04.777182 15687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:04 dist-test-slave-g0w1 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:04.780762 15687 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:10:04.782397 15687 webserver.cc:460] Webserver started at http://127.0.97.3:40785/ using document root <none> and password file <none>
I20260927 15:10:04.783214 15687 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:04.783375 15687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:04.789335 15687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:10:04.792685 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:04.794234 15687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:04.794472 15687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "7177d3bc750c40cb8fc36765bf2e1e71"
format_stamp: "Formatted at 2026-09-27 15:09:58 on dist-test-slave-g0w1"
server_key: "f9cf82a4a1020442357a530dadc557f8"
server_key_iv: "b127fb4c4062b208817d35b85a161ae3"
server_key_version: "encryptionkey@0"
I20260927 15:10:04.795671 15687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:04.815515 15687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:04.820617 15687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:04.820930 15687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:04.822008 15687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:04.823572 15687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:04.823746 15687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 15:10:04.823849 15687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:04.823904 15687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:04.860154 15687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260927 15:10:04.860198 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260927 15:10:04.861692 15687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:04.867578 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.865103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:54821)
0927 15:10:04.865990 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.866337 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":337,"threads_started":1}
W20260927 15:10:04.868598 15815 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41785 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41785: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
I20260927 15:10:04.868775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15687
I20260927 15:10:04.869622 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.869242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:40225)
0927 15:10:04.869421 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.869508 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:10:04.870769 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.870331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:50707)
0927 15:10:04.870554 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.870650 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 15:10:04.871033 15815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:10:04.885288 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.884492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:38845)
0927 15:10:04.884948 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.885039 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260927 15:10:05.384516 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:05.383467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:37545)
0927 15:10:05.384164 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:05.384265 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":411,"threads_started":1}
I20260927 15:10:05.872623 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:05.871758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:47361)
0927 15:10:05.872268 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:05.872369 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260927 15:10:05.887238 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:05.886289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:37813)
0927 15:10:05.886888 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:05.886988 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":318,"threads_started":1}
I20260927 15:10:06.386343 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:06.385576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:36841)
0927 15:10:06.386074 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:06.386163 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":252,"threads_started":1}
I20260927 15:10:06.874775 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:06.873858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:51471)
0927 15:10:06.874464 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:06.874575 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":280,"threads_started":1}
I20260927 15:10:06.889273 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:06.888355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:49171)
0927 15:10:06.888997 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:06.889096 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":394,"threads_started":1}
I20260927 15:10:07.388266 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:07.387481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:34341)
0927 15:10:07.387976 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:07.388067 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260927 15:10:07.876689 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:07.875911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:48399)
0927 15:10:07.876411 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:07.876503 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
I20260927 15:10:07.891389 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:07.890447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:41943)
0927 15:10:07.890899 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:07.890984 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":213,"threads_started":1}
I20260927 15:10:08.390419 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:08.389414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:47065)
0927 15:10:08.390109 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:08.390229 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":374,"threads_started":1}
I20260927 15:10:08.878474 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:08.877729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:38323)
0927 15:10:08.878211 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:08.878301 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
I20260927 15:10:08.893070 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:08.892330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:40781)
0927 15:10:08.892822 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:08.892901 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":233,"threads_started":1}
I20260927 15:10:09.392496 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:09.391636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:42205)
0927 15:10:09.392222 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:09.392303 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":306,"threads_started":1}
I20260927 15:10:09.880537 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:09.879738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:49659)
0927 15:10:09.880247 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:09.880343 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":242,"threads_started":1}
I20260927 15:10:09.894891 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:09.894010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:33465)
0927 15:10:09.894621 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:09.894714 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":344,"threads_started":1}
I20260927 15:10:10.394421 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:10.393543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:34859)
0927 15:10:10.394125 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:10.394253 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":310,"threads_started":1}
I20260927 15:10:10.882275 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:10.881514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:58025)
0927 15:10:10.881970 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:10.882063 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260927 15:10:10.896934 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:10.896040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:33475)
0927 15:10:10.896594 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:10.896704 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":265,"threads_started":1}
I20260927 15:10:11.396489 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:11.395545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:56153)
0927 15:10:11.396158 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:11.396260 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":252,"threads_started":1}
I20260927 15:10:11.884287 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:11.883435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:58179)
0927 15:10:11.883957 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:11.884057 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":239,"threads_started":1}
I20260927 15:10:11.899019 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:11.898089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:34815)
0927 15:10:11.898700 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:11.898780 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":309,"threads_started":1}
I20260927 15:10:12.398432 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:12.397612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:47345)
0927 15:10:12.398124 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:12.398211 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260927 15:10:12.885936 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:12.885237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:55045)
0927 15:10:12.885694 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:12.885775 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260927 15:10:12.900962 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:12.900136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:58779)
0927 15:10:12.900691 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:12.900783 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":264,"threads_started":1}
I20260927 15:10:13.400520 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:13.399571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:45759)
0927 15:10:13.400240 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:13.400336 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260927 15:10:13.887971 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:13.887011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:52135)
0927 15:10:13.887688 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:13.887792 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":465,"threads_started":1}
I20260927 15:10:13.902657 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:13.901971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:45607)
0927 15:10:13.902414 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:13.902494 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
I20260927 15:10:14.402645 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:14.401733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:54961)
0927 15:10:14.402359 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:14.402460 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":312,"threads_started":1}
I20260927 15:10:14.889983 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:14.889168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:36463)
0927 15:10:14.889706 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:14.889801 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260927 15:10:14.904516 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:14.903760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:38905)
0927 15:10:14.904229 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:14.904326 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":207,"threads_started":1}
I20260927 15:10:15.404666 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:15.403827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:38955)
0927 15:10:15.404364 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:15.404464 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":285,"threads_started":1}
I20260927 15:10:15.892205 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:15.891367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:59067)
0927 15:10:15.891909 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:15.892009 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":248,"threads_started":1}
I20260927 15:10:15.906306 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:15.905569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:48627)
0927 15:10:15.906051 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:15.906142 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":247,"threads_started":1}
I20260927 15:10:16.406581 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:16.405771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:37433)
0927 15:10:16.406284 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:16.406387 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":263,"threads_started":1}
I20260927 15:10:16.893939 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:16.893220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:47891)
0927 15:10:16.893684 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:16.893776 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260927 15:10:16.908197 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:16.907388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:54889)
0927 15:10:16.907870 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:16.907963 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":224,"threads_started":1}
I20260927 15:10:17.408692 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:17.407791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:52663)
0927 15:10:17.408388 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:17.408488 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":304,"threads_started":1}
I20260927 15:10:17.896040 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:17.895064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:54823)
0927 15:10:17.895659 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:17.895772 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":252,"threads_started":1}
I20260927 15:10:17.910133 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:17.909273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:45923)
0927 15:10:17.909848 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:17.909938 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":322,"threads_started":1}
I20260927 15:10:18.410889 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:18.410088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:43037)
0927 15:10:18.410609 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:18.410698 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":292,"threads_started":1}
I20260927 15:10:18.897927 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:18.897086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:40883)
0927 15:10:18.897643 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:18.897733 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
I20260927 15:10:18.912396 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:18.911538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:48067)
0927 15:10:18.912093 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:18.912193 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":310,"threads_started":1}
I20260927 15:10:19.413010 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.412261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:34849)
0927 15:10:19.412726 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.412819 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260927 15:10:19.881085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15391
I20260927 15:10:19.900243 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.899284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:38899)
0927 15:10:19.899870 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.899960 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":293,"threads_started":1}
I20260927 15:10:19.902782 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.882377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.0.97.1:41517)
0927 15:10:19.883104 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.883128 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.883593 (+ 465us) server_negotiation.cc:407] Connection header received
0927 15:10:19.885495 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.885506 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.886035 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.886377 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:19.888008 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.889493 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.891052 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.891881 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.892092 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.895584 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:19.895593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:19.895612 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:19.895678 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:19.898947 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.899768 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.899781 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.899793 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.899994 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.900559 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.900564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.900567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.901212 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:19.901570 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.902042 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.902496 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":294,"threads_started":1}
I20260927 15:10:19.904383 15548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:19.994568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15391
I20260927 15:10:20.012359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15556
I20260927 15:10:20.033777 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:20.013304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.0.97.2:37177)
0927 15:10:20.013887 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:20.013901 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:20.013938 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:10:20.015585 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:20.015598 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:20.016080 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:20.016470 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:20.018066 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:20.019508 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:20.020809 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:20.021549 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:20.021782 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:20.026651 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:20.026660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:20.026680 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:20.026735 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:20.029987 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:20.030825 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:20.030833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:20.030841 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:20.031052 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:20.031538 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:20.031544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:20.031547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:20.032156 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:20.032499 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:20.032996 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 15:10:20.033469 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":290,"threads_started":1}
I20260927 15:10:20.035399 15679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:20.122606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15556
I20260927 15:10:20.141552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15687
I20260927 15:10:20.161007 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:20.142724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.0.97.3:44267)
0927 15:10:20.143473 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:20.143489 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:20.143533 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:10:20.145187 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:20.145199 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:20.145671 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:20.145992 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:20.147508 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:20.148839 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:20.150186 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:20.150964 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:20.151223 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:20.153940 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:20.153948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:20.153966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:20.154010 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:20.157221 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:20.158081 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:20.158089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:20.158097 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:20.158308 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:20.158763 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:20.158768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:20.158770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:20.159548 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:20.159788 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:20.160214 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 15:10:20.160700 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":440,"threads_started":1}
I20260927 15:10:20.162479 15810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:20.247849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15687
2026-09-27T15:10:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25021 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:10:20 dist-test-slave-g0w1 krb5kdc[15873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:10:20 dist-test-slave-g0w1 krb5kdc[15873](info): set up 2 sockets
Sep 27 15:10:20 dist-test-slave-g0w1 krb5kdc[15873](info): commencing operation
krb5kdc: starting...
W20260927 15:10:21.368211 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:10:21 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:10:21.618865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40869
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:21.956441 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:21.956835 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:21.957051 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:21.968309 15886 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:10:21.968431 15886 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:10:21.968501 15886 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:10:21.968575 15886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:21.968657 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:21.968755 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:21.968837 15886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:21.968914 15886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:21.979741 15886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:21.981154 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:21.982570 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:21.996003 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:21.995930 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:21.995961 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:21.997326 15886 server_base.cc:1034] running on GCE node
I20260927 15:10:21.998188 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:21.999819 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:22.001116 15886 hybrid_clock.cc:648] HybridClock initialized: now 1790521822001022 us; error 50 us; skew 500 ppm
I20260927 15:10:22.001509 15886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:22 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:22.008900 15886 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:10:22.010598 15886 webserver.cc:460] Webserver started at http://127.0.97.62:38353/ using document root <none> and password file <none>
I20260927 15:10:22.011490 15886 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:22.011634 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:22.011961 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:22.015208 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "a157ac72fdc347e3b2800b09d3756c5c"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "64b27b6b8e907c4671e1a5a513c02c4d"
server_key_iv: "290f349ed70760c7374ab0ea9336710a"
server_key_version: "encryptionkey@0"
I20260927 15:10:22.016065 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "a157ac72fdc347e3b2800b09d3756c5c"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "64b27b6b8e907c4671e1a5a513c02c4d"
server_key_iv: "290f349ed70760c7374ab0ea9336710a"
server_key_version: "encryptionkey@0"
I20260927 15:10:22.021566 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:10:22.025290 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:22.026808 15886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:22.027118 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "a157ac72fdc347e3b2800b09d3756c5c"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "64b27b6b8e907c4671e1a5a513c02c4d"
server_key_iv: "290f349ed70760c7374ab0ea9336710a"
server_key_version: "encryptionkey@0"
I20260927 15:10:22.027393 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:22.072997 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:22.077803 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:22.078094 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:22.097139 15886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40869
I20260927 15:10:22.097164 15952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40869 every 8 connection(s)
I20260927 15:10:22.098595 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:22.102650 15953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:22.103550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15886
I20260927 15:10:22.103762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:10:22.104398 388 external_mini_cluster.cc:1428] Setting key 4e985141a4ba566c5bcb8f8f39ea0667
I20260927 15:10:22.117211 15953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap starting.
Sep 27 15:10:22 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:22.121469 15953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:22.122689 15953 log.cc:824] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:22.126191 15953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: No bootstrap required, opened a new log
I20260927 15:10:22.129967 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:22.106489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.0.97.62:40869)
0927 15:10:22.107353 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:22.107376 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:22.107417 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:10:22.109040 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:22.109067 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:22.109484 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:22.109836 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:22.111766 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:22.113149 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:22.115178 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:22.115956 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:22.116117 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:22.121457 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:22.121466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:22.121482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:22.121525 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:22.125209 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:22.126152 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:22.126160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:22.126167 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:22.126414 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:22.126895 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:22.126899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:22.126902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:22.127602 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:22.127835 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:22.128374 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 15:10:22.128863 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":373,"threads_started":1}
I20260927 15:10:22.135540 15953 raft_consensus.cc:348] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:22.135993 15953 raft_consensus.cc:374] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:22.136117 15953 raft_consensus.cc:729] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157ac72fdc347e3b2800b09d3756c5c, State: Initialized, Role: FOLLOWER
I20260927 15:10:22.136696 15953 consensus_queue.cc:260] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:22.136976 15953 raft_consensus.cc:388] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:22.137111 15953 raft_consensus.cc:482] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:22.137295 15953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:22.139674 15953 raft_consensus.cc:504] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:22.140166 15953 leader_election.cc:302] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a157ac72fdc347e3b2800b09d3756c5c; no voters:
I20260927 15:10:22.140771 15953 leader_election.cc:288] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:10:22.140929 15958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:22.141673 15958 raft_consensus.cc:686] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 LEADER]: Becoming Leader. State: Replica: a157ac72fdc347e3b2800b09d3756c5c, State: Running, Role: LEADER
I20260927 15:10:22.142241 15958 consensus_queue.cc:237] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:22.142887 15953 sys_catalog.cc:564] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:22.145296 15959 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a157ac72fdc347e3b2800b09d3756c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } } }
I20260927 15:10:22.145721 15959 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:22.146426 15960 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a157ac72fdc347e3b2800b09d3756c5c. Latest consensus state: current_term: 1 leader_uuid: "a157ac72fdc347e3b2800b09d3756c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } } }
I20260927 15:10:22.146842 15960 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:22.149190 15966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:22.158804 15966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:22.168952 15966 catalog_manager.cc:1347] Generated new cluster ID: 4d217da13bfe4a21a8eaa20f44d1afee
I20260927 15:10:22.169095 15966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:10:22.190435 15966 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:10:22.192019 15966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:10:22.204197 15966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Generated new TSK 0
I20260927 15:10:22.205139 15966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:10:22.274385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40869
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:22.626361 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:22.626678 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:22.626778 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:22.635222 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:22.635358 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:22.635486 15979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:10:22.645892 15979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:22.647317 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:22.648794 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:22.663602 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:22.663506 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:22.663561 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:22.664901 15979 server_base.cc:1034] running on GCE node
I20260927 15:10:22.665634 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:22.666712 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:22.668002 15979 hybrid_clock.cc:648] HybridClock initialized: now 1790521822667929 us; error 58 us; skew 500 ppm
I20260927 15:10:22.668385 15979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:22 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:22.672030 15979 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:10:22.673583 15979 webserver.cc:460] Webserver started at http://127.0.97.1:45317/ using document root <none> and password file <none>
I20260927 15:10:22.674283 15979 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:22.674388 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:22.674654 15979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:22.677871 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "8b2b562def804a969478a5da77a2fa0b"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "f5e859e88501cc3b78a01030c5298c34"
server_key_iv: "a279363403fde5fa3277490ad270e94a"
server_key_version: "encryptionkey@0"
I20260927 15:10:22.678632 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b2b562def804a969478a5da77a2fa0b"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "f5e859e88501cc3b78a01030c5298c34"
server_key_iv: "a279363403fde5fa3277490ad270e94a"
server_key_version: "encryptionkey@0"
I20260927 15:10:22.684268 15979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 15:10:22.688053 15994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:22.689604 15979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:22.689818 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "8b2b562def804a969478a5da77a2fa0b"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "f5e859e88501cc3b78a01030c5298c34"
server_key_iv: "a279363403fde5fa3277490ad270e94a"
server_key_version: "encryptionkey@0"
I20260927 15:10:22.690090 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:22.705319 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:22.710590 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:22.710947 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:22.712162 15979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:22.713639 15979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:22.713728 15979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:22.713852 15979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:22.713929 15979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:22.749768 15979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40059
I20260927 15:10:22.749811 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40059 every 8 connection(s)
I20260927 15:10:22.751216 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:22.759564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 15979
I20260927 15:10:22.759793 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:10:22.760458 388 external_mini_cluster.cc:1428] Setting key dfc273c2af2be611528a3a1aef03a61e
Sep 27 15:10:22 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:22.774601 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:22.754064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40523 (local address 127.0.97.62:40869)
0927 15:10:22.754560 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:22.754565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:22.755590 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 15:10:22.757542 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:22.757548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:22.757649 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:22.757774 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:22.759559 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:22.760459 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:22.763023 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:22.763513 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:22.763618 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:22.768405 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:22.768411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:22.768414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:22.768452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:22.771213 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:22.772170 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:22.772182 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:22.772185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:22.772309 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:22.772774 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:22.772778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:22.772780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:22.773049 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:22.773209 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:22.773969 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 15:10:22.774213 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
I20260927 15:10:22.775723 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:22.754596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:40523)
0927 15:10:22.755417 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:22.755457 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:22.757241 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:22.758038 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:22.758047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:22.758605 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:22.759360 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:22.759382 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:22.760745 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:22.760750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:22.762837 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:22.762859 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:22.763772 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:22.763783 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:22.764191 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:22.765024 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:22.765061 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:22.768065 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:22.771380 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:22.771392 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:22.771419 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:22.772023 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:22.772426 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:22.772431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:22.772435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:22.772642 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:22.773190 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:22.773199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:22.773574 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 15:10:22.773851 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 15:10:22.774447 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":281,"threads_started":1}
I20260927 15:10:22.777529 16107 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40869
I20260927 15:10:22.777876 16107 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:22.778661 16107 heartbeater.cc:511] Master 127.0.97.62:40869 requested a full tablet report, sending...
I20260927 15:10:22.781273 15918 ts_manager.cc:194] Registered new tserver with Master: 8b2b562def804a969478a5da77a2fa0b (127.0.97.1:40059)
I20260927 15:10:22.783793 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40523
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:10:22.818837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40869
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:23.167337 16113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:23.167704 16113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:23.167883 16113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:23.175575 16113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:23.175765 16113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:23.175943 16113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:10:23.186410 16113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:23.187968 16113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:23.189340 16113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:23.203603 16121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:23.203543 16119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:23.203606 16118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:23.209985 16113 server_base.cc:1034] running on GCE node
I20260927 15:10:23.210778 16113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:23.212112 16113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:23.213379 16113 hybrid_clock.cc:648] HybridClock initialized: now 1790521823213323 us; error 68 us; skew 500 ppm
I20260927 15:10:23.213743 16113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:23 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:23.217422 16113 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:10:23.219290 16113 webserver.cc:460] Webserver started at http://127.0.97.2:35503/ using document root <none> and password file <none>
I20260927 15:10:23.220064 16113 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:23.220214 16113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:23.220539 16113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:23.223877 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "fed23c9f59f2401a990718e31f3fc4fe"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "35955ddd3e1b15a422876894469729e1"
server_key_iv: "a09aab771edd42c80ddcc83a8523e192"
server_key_version: "encryptionkey@0"
I20260927 15:10:23.224727 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fed23c9f59f2401a990718e31f3fc4fe"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "35955ddd3e1b15a422876894469729e1"
server_key_iv: "a09aab771edd42c80ddcc83a8523e192"
server_key_version: "encryptionkey@0"
I20260927 15:10:23.230695 16113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:10:23.234727 16128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:23.236411 16113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:10:23.236634 16113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "fed23c9f59f2401a990718e31f3fc4fe"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "35955ddd3e1b15a422876894469729e1"
server_key_iv: "a09aab771edd42c80ddcc83a8523e192"
server_key_version: "encryptionkey@0"
I20260927 15:10:23.236871 16113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:23.255769 16113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:23.260408 16113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:23.260715 16113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:23.261720 16113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:23.263213 16113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:23.263340 16113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:23.263482 16113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:23.263559 16113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:23.298089 16113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37693
I20260927 15:10:23.298141 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37693 every 8 connection(s)
I20260927 15:10:23.299598 16113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:23.304046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16113
I20260927 15:10:23.304306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:10:23.304912 388 external_mini_cluster.cc:1428] Setting key 1fbf77f714313f8e08ad42be6cbd03cb
Sep 27 15:10:23 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:23.325894 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.302592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50715 (local address 127.0.97.62:40869)
0927 15:10:23.303161 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:23.303166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:23.304279 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 15:10:23.306617 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:23.306623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:23.306711 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:23.306852 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:23.309192 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.309918 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.312801 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.313273 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.313396 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.317922 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:23.317933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:23.317941 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:23.317995 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:23.321849 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:23.322776 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:23.322785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:23.322791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:23.322961 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:23.323612 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:23.323620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:23.323626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:23.324038 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:23.324217 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:23.325092 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 15:10:23.325385 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260927 15:10:23.326642 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.303037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:50715)
0927 15:10:23.303977 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:23.304044 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:23.306305 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:23.307147 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:23.307160 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:23.308057 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:23.308979 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.309009 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.310084 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.310093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.312425 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.312444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.313542 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.313551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.313957 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.314801 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:23.314838 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:23.317592 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:23.322051 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:23.322058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:23.322075 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:23.322574 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:23.323238 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:23.323241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:23.323244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:23.323455 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:23.324249 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:23.324256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:23.324646 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 15:10:23.324905 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 15:10:23.325379 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":289,"threads_started":1}
I20260927 15:10:23.328518 16241 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40869
I20260927 15:10:23.328914 16241 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:23.329998 16241 heartbeater.cc:511] Master 127.0.97.62:40869 requested a full tablet report, sending...
I20260927 15:10:23.332095 15918 ts_manager.cc:194] Registered new tserver with Master: fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:10:23.333245 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:10:23.364705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40869
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:23.707422 16247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:23.707820 16247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:23.708019 16247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:23.715751 16247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:23.715956 16247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:23.716163 16247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:10:23.728263 16247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:23.729717 16247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:23.731360 16247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:23.746543 16252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:23.746599 16255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:23.746625 16253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:23.747483 16247 server_base.cc:1034] running on GCE node
I20260927 15:10:23.748246 16247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:23.749642 16247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:23.750907 16247 hybrid_clock.cc:648] HybridClock initialized: now 1790521823750833 us; error 75 us; skew 500 ppm
I20260927 15:10:23.751345 16247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:23 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:23.754859 16247 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:10:23.756523 16247 webserver.cc:460] Webserver started at http://127.0.97.3:46571/ using document root <none> and password file <none>
I20260927 15:10:23.757289 16247 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:23.757431 16247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:23.757838 16247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:23.761062 16247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "98982e61e7534fd98736e493fde46c3c"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "34ec7cd6cece09e0611b31185dd81f82"
server_key_iv: "3729f7e56f8c22db75f93e19ecd30d1b"
server_key_version: "encryptionkey@0"
I20260927 15:10:23.761934 16247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "98982e61e7534fd98736e493fde46c3c"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "34ec7cd6cece09e0611b31185dd81f82"
server_key_iv: "3729f7e56f8c22db75f93e19ecd30d1b"
server_key_version: "encryptionkey@0"
I20260927 15:10:23.768220 16247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 15:10:23.772660 16262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:23.774501 16247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:10:23.774729 16247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "98982e61e7534fd98736e493fde46c3c"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "34ec7cd6cece09e0611b31185dd81f82"
server_key_iv: "3729f7e56f8c22db75f93e19ecd30d1b"
server_key_version: "encryptionkey@0"
I20260927 15:10:23.774966 16247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:23.787663 16107 heartbeater.cc:503] Master 127.0.97.62:40869 was elected leader, sending a full tablet report...
I20260927 15:10:23.790031 16247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:23.794999 16247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:23.795368 16247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:23.796464 16247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:23.798027 16247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:23.798146 16247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:23.798316 16247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:23.798398 16247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:23.835552 16247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38881
I20260927 15:10:23.835616 16374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38881 every 8 connection(s)
I20260927 15:10:23.837061 16247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:23.839617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16247
I20260927 15:10:23.839802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:10:23.840405 388 external_mini_cluster.cc:1428] Setting key 1ec656fce4e423ca4b311b3277f235a8
Sep 27 15:10:23 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:23.860496 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.840021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36215 (local address 127.0.97.62:40869)
0927 15:10:23.840593 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:23.840600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:23.842058 (+ 1458us) server_negotiation.cc:407] Connection header received
0927 15:10:23.844287 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:23.844292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:23.844373 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:23.844466 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:23.846087 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.846883 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.848583 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.849030 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.849177 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.853684 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:23.853690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:23.853693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:23.853732 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:23.857010 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:23.857893 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:23.857897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:23.857900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:23.858101 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:23.858584 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:23.858588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:23.858590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:23.858891 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:23.859040 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:23.859957 (+ 917us) server_negotiation.cc:299] Negotiation successful
0927 15:10:23.860213 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
I20260927 15:10:23.861442 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.840758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:36215)
0927 15:10:23.841708 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:23.841761 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:23.843973 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:23.844705 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:23.844716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:23.845278 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:23.845892 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.845910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.847027 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.847031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.848447 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.848465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.849322 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.849328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.849601 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.850390 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:23.850429 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:23.853357 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:23.857168 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:23.857177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:23.857193 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:23.857713 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:23.858231 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:23.858235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:23.858237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:23.858445 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:23.859061 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:23.859102 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:23.859549 (+ 447us) client_negotiation.cc:769] Sending connection context
0927 15:10:23.859798 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 15:10:23.860286 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":404,"threads_started":1}
I20260927 15:10:23.863051 16375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40869
I20260927 15:10:23.863515 16375 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:23.864326 16375 heartbeater.cc:511] Master 127.0.97.62:40869 requested a full tablet report, sending...
I20260927 15:10:23.866268 15918 ts_manager.cc:194] Registered new tserver with Master: 98982e61e7534fd98736e493fde46c3c (127.0.97.3:38881)
I20260927 15:10:23.867446 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36215
I20260927 15:10:23.869504 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:23.888495 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.874526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.0.97.62:40869)
0927 15:10:23.874770 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:23.874779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:23.875231 (+ 452us) server_negotiation.cc:407] Connection header received
0927 15:10:23.875399 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:23.875404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:23.875484 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:23.875599 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:23.877127 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.877869 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.879438 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.879869 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.880021 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.882632 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:23.882638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:23.882642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:23.882681 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:23.885339 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:23.886151 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:23.886157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:23.886161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:23.886300 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:23.886751 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:23.886757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:23.886762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:23.887250 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:23.887517 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:23.887870 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:10:23.888158 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:10:23.893507 15917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:10:23.900662 15917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:23.928421 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.913792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38881 (local address 127.0.0.1:38646)
0927 15:10:23.914863 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:23.914923 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:23.915189 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:23.917425 (+ 2236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:23.917438 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:23.917487 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:23.918085 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.918102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.921174 (+ 3072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.921183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.924646 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.924678 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.927267 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.927275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.927543 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.927616 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 15:10:23.927791 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:10:23.928014 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":1091,"threads_started":1}
I20260927 15:10:23.929491 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.914269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.0.97.3:38881)
0927 15:10:23.916147 (+ 1878us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:23.916161 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:23.916203 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:10:23.916327 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:23.916339 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:23.916685 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:23.918323 (+ 1638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:23.918695 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.920901 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.924974 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.926727 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.926850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.927130 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:23.928957 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0927 15:10:23.929168 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1679,"thread_start_us":1527,"threads_started":1}
I20260927 15:10:23.935181 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.913982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37693 (local address 127.0.0.1:55370)
0927 15:10:23.924729 (+ 10747us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:23.924751 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:23.924893 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:23.927163 (+ 2270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:23.927170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:23.927202 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:23.927668 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.927679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.930681 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.930690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.932371 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.932385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.934321 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.934328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.934511 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.934553 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:10:23.934622 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:10:23.934770 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10582,"thread_start_us":16264,"threads_started":1}
I20260927 15:10:23.934540 16310 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8dfe57e7be48308f6c21f12146faee (DEFAULT_TABLE table=test-table [id=47e130a22a1b4057a381962363e4f580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:23.936319 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.914764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.0.97.2:37693)
0927 15:10:23.926121 (+ 11357us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:23.926144 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:23.926189 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:10:23.926303 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:23.926312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:23.926607 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:23.926865 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:23.928085 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.930492 (+ 2407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.932540 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.934070 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.935563 (+ 1493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.935796 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:23.935856 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 15:10:23.935990 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11176,"thread_start_us":13050,"threads_started":1}
I20260927 15:10:23.936632 16310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8dfe57e7be48308f6c21f12146faee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:23.940120 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.916408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40059 (local address 127.0.0.1:58510)
0927 15:10:23.929873 (+ 13465us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:23.929893 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:23.930001 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:23.933845 (+ 3844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:23.933850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:23.933885 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:23.934057 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.934067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.936158 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.936164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.937808 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:23.937823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.939266 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.939273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:23.939476 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.939522 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:10:23.939648 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:10:23.939803 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13352,"thread_start_us":7974,"threads_started":1}
I20260927 15:10:23.940167 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:23.917256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.0.97.1:40059)
0927 15:10:23.933112 (+ 15856us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:23.933121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:23.933151 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:10:23.933251 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:23.933260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:23.933540 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:23.933843 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:23.934268 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.935983 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.937987 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:23.938997 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:23.939217 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:23.939467 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:23.939689 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:10:23.939859 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15661,"thread_start_us":15494,"threads_started":1}
I20260927 15:10:23.940629 16176 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8dfe57e7be48308f6c21f12146faee (DEFAULT_TABLE table=test-table [id=47e130a22a1b4057a381962363e4f580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:23.942483 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8dfe57e7be48308f6c21f12146faee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:23.945655 16042 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8dfe57e7be48308f6c21f12146faee (DEFAULT_TABLE table=test-table [id=47e130a22a1b4057a381962363e4f580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:23.946923 16042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8dfe57e7be48308f6c21f12146faee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:23.962231 16394 tablet_bootstrap.cc:492] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Bootstrap starting.
I20260927 15:10:23.963259 16396 tablet_bootstrap.cc:492] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Bootstrap starting.
I20260927 15:10:23.967854 16395 tablet_bootstrap.cc:492] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Bootstrap starting.
I20260927 15:10:23.967965 16394 tablet_bootstrap.cc:654] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:23.969431 16394 log.cc:824] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:23.971498 16396 tablet_bootstrap.cc:654] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:23.972167 16395 tablet_bootstrap.cc:654] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:23.972788 16396 log.cc:824] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:23.973773 16395 log.cc:824] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:23.978152 16396 tablet_bootstrap.cc:492] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: No bootstrap required, opened a new log
I20260927 15:10:23.978381 16394 tablet_bootstrap.cc:492] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: No bootstrap required, opened a new log
I20260927 15:10:23.978381 16395 tablet_bootstrap.cc:492] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: No bootstrap required, opened a new log
I20260927 15:10:23.978502 16396 ts_tablet_manager.cc:1397] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20260927 15:10:23.978678 16395 ts_tablet_manager.cc:1397] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260927 15:10:23.978672 16394 ts_tablet_manager.cc:1397] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260927 15:10:23.986840 16396 raft_consensus.cc:348] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:23.987372 16396 raft_consensus.cc:374] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:23.987473 16396 raft_consensus.cc:729] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed23c9f59f2401a990718e31f3fc4fe, State: Initialized, Role: FOLLOWER
I20260927 15:10:23.988209 16396 consensus_queue.cc:260] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:23.990044 16394 raft_consensus.cc:348] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:23.990046 16395 raft_consensus.cc:348] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:23.990537 16394 raft_consensus.cc:374] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:23.990537 16395 raft_consensus.cc:374] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:23.990656 16395 raft_consensus.cc:729] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2b562def804a969478a5da77a2fa0b, State: Initialized, Role: FOLLOWER
I20260927 15:10:23.990649 16394 raft_consensus.cc:729] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98982e61e7534fd98736e493fde46c3c, State: Initialized, Role: FOLLOWER
I20260927 15:10:23.991509 16241 heartbeater.cc:503] Master 127.0.97.62:40869 was elected leader, sending a full tablet report...
I20260927 15:10:23.991510 16395 consensus_queue.cc:260] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:23.992303 16396 ts_tablet_manager.cc:1428] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260927 15:10:23.992146 16394 consensus_queue.cc:260] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:23.993909 16375 heartbeater.cc:503] Master 127.0.97.62:40869 was elected leader, sending a full tablet report...
I20260927 15:10:23.995278 16394 ts_tablet_manager.cc:1428] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260927 15:10:23.998974 16395 ts_tablet_manager.cc:1428] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
W20260927 15:10:24.003700 16108 tablet.cc:2369] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:10:24.051832 16242 tablet.cc:2369] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:24.074867 16402 raft_consensus.cc:482] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:24.075269 16402 raft_consensus.cc:504] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:24.077100 16402 leader_election.cc:288] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693), 8b2b562def804a969478a5da77a2fa0b (127.0.97.1:40059)
I20260927 15:10:24.084174 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.077428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37693 (local address 127.0.97.3:52647)
0927 15:10:24.077912 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:24.077936 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:24.078126 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:24.078612 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:24.078620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:24.078752 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:24.078957 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:24.078971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.080282 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.080288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:24.081984 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:24.081998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.083299 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.083306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:24.083516 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.083552 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:10:24.083661 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:10:24.083847 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":39,"spinlock_wait_cycles":46848}
I20260927 15:10:24.084196 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.077658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52647 (local address 127.0.97.2:37693)
0927 15:10:24.077858 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:24.077867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:24.078040 (+ 173us) server_negotiation.cc:407] Connection header received
0927 15:10:24.078293 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:24.078302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:24.078391 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:24.078504 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:24.079147 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.080128 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.082174 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.083114 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.083243 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.083318 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:24.083708 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 15:10:24.083854 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:10:24.084686 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.077618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58775 (local address 127.0.97.1:40059)
0927 15:10:24.077844 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:24.077852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:24.078646 (+ 794us) server_negotiation.cc:407] Connection header received
0927 15:10:24.078845 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:24.078850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:24.078935 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:24.079061 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:24.079696 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.080560 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.082451 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.083560 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.083690 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.083788 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:24.084274 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:10:24.084426 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:10:24.085114 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.077651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40059 (local address 127.0.97.3:58775)
0927 15:10:24.078508 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:24.078533 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:24.078691 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:24.079127 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:24.079131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:24.079155 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:24.079517 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:24.079528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.080711 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.080715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:24.082294 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:24.082309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.083803 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.083816 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:24.084019 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.084071 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:10:24.084234 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:10:24.084728 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":2,"thread_start_us":467,"threads_started":1}
I20260927 15:10:24.085470 16196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dfe57e7be48308f6c21f12146faee" candidate_uuid: "98982e61e7534fd98736e493fde46c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed23c9f59f2401a990718e31f3fc4fe" is_pre_election: true
I20260927 15:10:24.086063 16196 raft_consensus.cc:2454] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98982e61e7534fd98736e493fde46c3c in term 0.
I20260927 15:10:24.086127 16062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dfe57e7be48308f6c21f12146faee" candidate_uuid: "98982e61e7534fd98736e493fde46c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2b562def804a969478a5da77a2fa0b" is_pre_election: true
I20260927 15:10:24.086709 16062 raft_consensus.cc:2454] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98982e61e7534fd98736e493fde46c3c in term 0.
I20260927 15:10:24.086877 16265 leader_election.cc:302] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98982e61e7534fd98736e493fde46c3c, fed23c9f59f2401a990718e31f3fc4fe; no voters:
I20260927 15:10:24.087363 16402 raft_consensus.cc:2781] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:24.087512 16402 raft_consensus.cc:482] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:24.087620 16402 raft_consensus.cc:3037] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Advancing to term 1
W20260927 15:10:24.088810 16376 tablet.cc:2369] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:24.090423 16402 raft_consensus.cc:504] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:24.091068 16402 leader_election.cc:288] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [CANDIDATE]: Term 1 election: Requested vote from peers fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693), 8b2b562def804a969478a5da77a2fa0b (127.0.97.1:40059)
I20260927 15:10:24.091605 16196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dfe57e7be48308f6c21f12146faee" candidate_uuid: "98982e61e7534fd98736e493fde46c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed23c9f59f2401a990718e31f3fc4fe"
I20260927 15:10:24.091619 16062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8dfe57e7be48308f6c21f12146faee" candidate_uuid: "98982e61e7534fd98736e493fde46c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2b562def804a969478a5da77a2fa0b"
I20260927 15:10:24.091809 16196 raft_consensus.cc:3037] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:24.091835 16062 raft_consensus.cc:3037] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:24.094161 16196 raft_consensus.cc:2454] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98982e61e7534fd98736e493fde46c3c in term 1.
I20260927 15:10:24.094195 16062 raft_consensus.cc:2454] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98982e61e7534fd98736e493fde46c3c in term 1.
I20260927 15:10:24.094626 16265 leader_election.cc:302] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98982e61e7534fd98736e493fde46c3c, fed23c9f59f2401a990718e31f3fc4fe; no voters:
I20260927 15:10:24.094971 16402 raft_consensus.cc:2781] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:24.095443 16402 raft_consensus.cc:686] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 1 LEADER]: Becoming Leader. State: Replica: 98982e61e7534fd98736e493fde46c3c, State: Running, Role: LEADER
I20260927 15:10:24.095916 16402 consensus_queue.cc:237] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } }
I20260927 15:10:24.101017 15917 catalog_manager.cc:5495] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 98982e61e7534fd98736e493fde46c3c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "98982e61e7534fd98736e493fde46c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20260927 15:10:24.136349 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.130688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.0.97.3:38881)
0927 15:10:24.130888 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:24.130893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:24.130914 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:10:24.131053 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:24.131057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:24.131203 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:24.131378 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:24.131769 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.132622 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.134120 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.134517 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.134653 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.135222 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:24.135751 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:24.135921 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:24.135985 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:10:24.136069 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:10:24.145429 16062 raft_consensus.cc:1261] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Refusing update from remote peer 98982e61e7534fd98736e493fde46c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:24.145429 16196 raft_consensus.cc:1261] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Refusing update from remote peer 98982e61e7534fd98736e493fde46c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:24.146543 16404 consensus_queue.cc:1035] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:24.146956 16402 consensus_queue.cc:1035] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:24.161021 16407 mvcc.cc:205] Tried to move back new op lower bound from 7333977391688024064 to 7333977391500951552. Current Snapshot: MvccSnapshot[applied={T|T < 7333977391688024064}]
I20260927 15:10:24.172467 15917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:24.172793 15917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:24.176371 15917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Sending DeleteTablet for 3 replicas of tablet 8a8dfe57e7be48308f6c21f12146faee
I20260927 15:10:24.177713 16042 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8dfe57e7be48308f6c21f12146faee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58510
I20260927 15:10:24.177734 16310 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8dfe57e7be48308f6c21f12146faee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38646
I20260927 15:10:24.178352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15979
I20260927 15:10:24.178520 16176 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8dfe57e7be48308f6c21f12146faee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55370
I20260927 15:10:24.188755 16419 tablet_replica.cc:330] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: stopping tablet replica
I20260927 15:10:24.190219 16419 raft_consensus.cc:2229] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:24.191622 16419 raft_consensus.cc:2258] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:24.192292 16418 tablet_replica.cc:330] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: stopping tablet replica
I20260927 15:10:24.192948 16418 raft_consensus.cc:2229] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:24.195520 16417 tablet_replica.cc:330] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: stopping tablet replica
I20260927 15:10:24.195606 16418 raft_consensus.cc:2258] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:24.196049 16417 raft_consensus.cc:2229] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:24.197479 16419 ts_tablet_manager.cc:1905] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:24.197786 16418 ts_tablet_manager.cc:1905] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:24.196825 16417 raft_consensus.cc:2258] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 15:10:24 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:10:24.206337 16419 ts_tablet_manager.cc:1918] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:24.206539 16419 log.cc:1196] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/8a8dfe57e7be48308f6c21f12146faee
I20260927 15:10:24.206831 16418 ts_tablet_manager.cc:1918] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:24.207140 16419 ts_tablet_manager.cc:1939] T 8a8dfe57e7be48308f6c21f12146faee P fed23c9f59f2401a990718e31f3fc4fe: Deleting consensus metadata
I20260927 15:10:24.207144 16418 log.cc:1196] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/8a8dfe57e7be48308f6c21f12146faee
I20260927 15:10:24.207202 16417 ts_tablet_manager.cc:1905] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:24.207872 16418 ts_tablet_manager.cc:1939] T 8a8dfe57e7be48308f6c21f12146faee P 8b2b562def804a969478a5da77a2fa0b: Deleting consensus metadata
I20260927 15:10:24.209251 15904 catalog_manager.cc:4841] TS fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693): tablet 8a8dfe57e7be48308f6c21f12146faee (table test-table [id=47e130a22a1b4057a381962363e4f580]) successfully deleted
I20260927 15:10:24.209841 15903 catalog_manager.cc:4841] TS 8b2b562def804a969478a5da77a2fa0b (127.0.97.1:40059): tablet 8a8dfe57e7be48308f6c21f12146faee (table test-table [id=47e130a22a1b4057a381962363e4f580]) successfully deleted
I20260927 15:10:24.214350 16417 ts_tablet_manager.cc:1918] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:24.214618 16417 log.cc:1196] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/8a8dfe57e7be48308f6c21f12146faee
I20260927 15:10:24.215312 16417 ts_tablet_manager.cc:1939] T 8a8dfe57e7be48308f6c21f12146faee P 98982e61e7534fd98736e493fde46c3c: Deleting consensus metadata
I20260927 15:10:24.217190 15902 catalog_manager.cc:4841] TS 98982e61e7534fd98736e493fde46c3c (127.0.97.3:38881): tablet 8a8dfe57e7be48308f6c21f12146faee (table test-table [id=47e130a22a1b4057a381962363e4f580]) successfully deleted
I20260927 15:10:24.218220 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.179427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.97.1:40059)
0927 15:10:24.179870 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:24.179880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:24.189185 (+ 9305us) server_negotiation.cc:407] Connection header received
0927 15:10:24.189792 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:24.189798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:24.189906 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:24.189985 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:24.192426 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.193643 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.197950 (+ 4307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.198641 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.198760 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.209882 (+ 11122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:24.209898 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:24.209917 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:24.209976 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:24.214304 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:24.215552 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:24.215563 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:24.215566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:24.215762 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:24.216253 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:24.216258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:24.216262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:24.216755 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:24.217091 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:24.217531 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 15:10:24.217796 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260927 15:10:24.218933 16102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:24.319219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15979
I20260927 15:10:24.345019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 16113
Sep 27 15:10:24 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:10:24.363405 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.345917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.0.97.2:37693)
0927 15:10:24.346114 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:24.346120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:24.346152 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:10:24.346327 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:24.346334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:24.346439 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:24.346554 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:24.348154 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.349322 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.350894 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.351592 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.351729 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.356064 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:24.356086 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:24.356107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:24.356166 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:24.360108 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:24.360957 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:24.360967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:24.360971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:24.361147 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:24.361540 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:24.361547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:24.361551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:24.362043 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:24.362360 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:24.362656 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:10:24.362911 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:10:24.364291 16236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:24.458990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 16113
I20260927 15:10:24.484547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 16247
Sep 27 15:10:24 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:10:24.505523 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:24.485403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.0.97.3:38881)
0927 15:10:24.485571 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:24.485576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:24.485600 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:10:24.485816 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:24.485820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:24.485886 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:24.485975 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:24.487978 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.488883 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.491005 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:24.492013 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:24.492220 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:24.498722 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:24.498737 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:24.498755 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:24.498806 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:24.502183 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:24.503308 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:24.503317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:24.503319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:24.503486 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:24.503951 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:24.503954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:24.503957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:24.504331 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:24.504596 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:24.504956 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:10:24.505150 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:10:24.506361 16370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:24.631455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 16247
I20260927 15:10:24.651958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 15886
I20260927 15:10:24.653271 15948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:24.767596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 15886
I20260927 15:10:44.792802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40869
--webserver_interface=127.0.97.62
--webserver_port=38353
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:45.143378 16425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:45.143682 16425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:45.143857 16425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:45.151464 16425 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:10:45.151577 16425 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:10:45.151644 16425 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:10:45.151737 16425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:45.151823 16425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:45.151934 16425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:45.151994 16425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:45.152091 16425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:45.162434 16425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:45.163952 16425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:45.165333 16425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:45.190598 16434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:45.190653 16432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:45.190672 16431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:45.191033 16425 server_base.cc:1034] running on GCE node
I20260927 15:10:45.192378 16425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:45.194144 16425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:45.195434 16425 hybrid_clock.cc:648] HybridClock initialized: now 1790521845195346 us; error 52 us; skew 500 ppm
I20260927 15:10:45.195926 16425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:45 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:45.207767 16425 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:10:45.209854 16425 webserver.cc:460] Webserver started at http://127.0.97.62:38353/ using document root <none> and password file <none>
I20260927 15:10:45.210770 16425 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:45.210959 16425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:45.217775 16425 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:10:45.221251 16441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:45.222915 16425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:45.223326 16425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "a157ac72fdc347e3b2800b09d3756c5c"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "64b27b6b8e907c4671e1a5a513c02c4d"
server_key_iv: "290f349ed70760c7374ab0ea9336710a"
server_key_version: "encryptionkey@0"
I20260927 15:10:45.224614 16425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:45.249646 16425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:45.254925 16425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:45.255317 16425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:45.275795 16425 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40869
I20260927 15:10:45.275816 16492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40869 every 8 connection(s)
I20260927 15:10:45.277437 16425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:45.280057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16425
I20260927 15:10:45.281064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40059
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40869
--webserver_port=45317
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:45.288524 16493 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:10:45.292954 16493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap starting.
I20260927 15:10:45.314226 16493 log.cc:824] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:45.327008 16493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:45.327781 16493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap complete.
I20260927 15:10:45.337973 16493 raft_consensus.cc:348] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:45.338913 16493 raft_consensus.cc:729] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157ac72fdc347e3b2800b09d3756c5c, State: Initialized, Role: FOLLOWER
I20260927 15:10:45.339748 16493 consensus_queue.cc:260] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:45.340021 16493 raft_consensus.cc:388] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:45.340175 16493 raft_consensus.cc:482] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:45.340376 16493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:10:45.344974 16493 raft_consensus.cc:504] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:45.345626 16493 leader_election.cc:302] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a157ac72fdc347e3b2800b09d3756c5c; no voters:
I20260927 15:10:45.346405 16493 leader_election.cc:288] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:10:45.346590 16497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:10:45.347882 16497 raft_consensus.cc:686] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 LEADER]: Becoming Leader. State: Replica: a157ac72fdc347e3b2800b09d3756c5c, State: Running, Role: LEADER
I20260927 15:10:45.348661 16497 consensus_queue.cc:237] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:10:45.349032 16493 sys_catalog.cc:564] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:45.351828 16499 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a157ac72fdc347e3b2800b09d3756c5c. Latest consensus state: current_term: 2 leader_uuid: "a157ac72fdc347e3b2800b09d3756c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } } }
I20260927 15:10:45.352252 16499 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:45.352717 16498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a157ac72fdc347e3b2800b09d3756c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } } }
I20260927 15:10:45.353030 16498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:45.362464 16503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:45.371695 16503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:45.373361 16503 catalog_manager.cc:1259] Loaded cluster ID: 4d217da13bfe4a21a8eaa20f44d1afee
I20260927 15:10:45.373564 16503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:10:45.378014 16503 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:10:45.380357 16503 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Loaded TSK: 0
I20260927 15:10:45.413357 16503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Generated new TSK 1
I20260927 15:10:45.413803 16503 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:10:45.693447 16495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:45.693763 16495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:45.693918 16495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:45.701604 16495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:45.701758 16495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:45.701949 16495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:10:45.712781 16495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45317
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:45.714217 16495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:45.715758 16495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:45.730012 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:45.730031 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:45.730904 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:45.731984 16495 server_base.cc:1034] running on GCE node
I20260927 15:10:45.732815 16495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:45.734196 16495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:45.735481 16495 hybrid_clock.cc:648] HybridClock initialized: now 1790521845735406 us; error 68 us; skew 500 ppm
I20260927 15:10:45.735913 16495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:45 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:45.740145 16495 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:10:45.742012 16495 webserver.cc:460] Webserver started at http://127.0.97.1:45317/ using document root <none> and password file <none>
I20260927 15:10:45.742784 16495 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:45.742964 16495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:45.749475 16495 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:10:45.752925 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:45.754662 16495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:10:45.754892 16495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "8b2b562def804a969478a5da77a2fa0b"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "f5e859e88501cc3b78a01030c5298c34"
server_key_iv: "a279363403fde5fa3277490ad270e94a"
server_key_version: "encryptionkey@0"
I20260927 15:10:45.756043 16495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:45.776543 16495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:45.781438 16495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:45.781738 16495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:45.782797 16495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:45.784377 16495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:45.784490 16495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:45.784581 16495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:45.784638 16495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:45.821048 16495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40059
I20260927 15:10:45.821110 16642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40059 every 8 connection(s)
I20260927 15:10:45.822592 16495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:45.831055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16495
I20260927 15:10:45.831878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37693
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40869
--webserver_port=35503
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:10:45 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:45.851337 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:45.825696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58015 (local address 127.0.97.62:40869)
0927 15:10:45.826337 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:45.826359 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:45.826927 (+ 568us) server_negotiation.cc:407] Connection header received
0927 15:10:45.829215 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:45.829229 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:45.829715 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:45.830095 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:45.832333 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:45.834619 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:45.836560 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:45.837779 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:45.838023 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:45.842540 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:45.842549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:45.842567 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:45.842609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:45.845805 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:45.846946 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:45.846954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:45.846960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:45.847272 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:45.847881 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:45.847884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:45.847886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:45.848441 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:45.848604 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:45.849686 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0927 15:10:45.850098 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":242,"threads_started":1}
I20260927 15:10:45.851717 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:45.825958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:58015)
0927 15:10:45.826748 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:45.826786 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:45.828611 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:45.830270 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:45.830278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:45.831176 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:45.832032 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:45.832066 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:45.834856 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:45.834861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:45.836355 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:45.836374 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:45.838030 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:45.838037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:45.838276 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:45.839186 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:45.839213 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:45.842174 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:45.845999 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:45.846012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:45.846034 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:45.846769 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:45.847426 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:45.847434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:45.847441 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:45.847748 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:45.848605 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:45.848616 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:45.849114 (+ 498us) client_negotiation.cc:769] Sending connection context
0927 15:10:45.849438 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 15:10:45.850141 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":316,"threads_started":1}
I20260927 15:10:45.854907 16643 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40869
I20260927 15:10:45.855288 16643 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:45.856071 16643 heartbeater.cc:511] Master 127.0.97.62:40869 requested a full tablet report, sending...
I20260927 15:10:45.858296 16458 ts_manager.cc:194] Registered new tserver with Master: 8b2b562def804a969478a5da77a2fa0b (127.0.97.1:40059)
I20260927 15:10:45.860220 16458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58015
W20260927 15:10:46.186853 16647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:46.187294 16647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:46.187470 16647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:46.194973 16647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:46.195205 16647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:46.195394 16647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:10:46.205649 16647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35503
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:46.207136 16647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:46.208513 16647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:46.222779 16652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:46.222734 16653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:46.222811 16655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:46.223502 16647 server_base.cc:1034] running on GCE node
I20260927 15:10:46.224264 16647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:46.225272 16647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:46.226511 16647 hybrid_clock.cc:648] HybridClock initialized: now 1790521846226459 us; error 36 us; skew 500 ppm
I20260927 15:10:46.226914 16647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:46 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:46.230466 16647 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:10:46.232205 16647 webserver.cc:460] Webserver started at http://127.0.97.2:35503/ using document root <none> and password file <none>
I20260927 15:10:46.232997 16647 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:46.233168 16647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:46.239226 16647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:10:46.243922 16662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:46.245626 16647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:46.245865 16647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "fed23c9f59f2401a990718e31f3fc4fe"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "35955ddd3e1b15a422876894469729e1"
server_key_iv: "a09aab771edd42c80ddcc83a8523e192"
server_key_version: "encryptionkey@0"
I20260927 15:10:46.247007 16647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:46.262723 16647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:46.267804 16647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:46.268129 16647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:46.269215 16647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:46.270740 16647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:46.270889 16647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:46.271034 16647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:46.271158 16647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:10:46.306430 16647 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37693
I20260927 15:10:46.306442 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37693 every 8 connection(s)
I20260927 15:10:46.308254 16647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:46.311506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16647
I20260927 15:10:46.312258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38881
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40869
--webserver_port=46571
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:10:46 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:46.332722 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.311149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50731 (local address 127.0.97.62:40869)
0927 15:10:46.311375 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.311382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.312763 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 15:10:46.315827 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.315832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.315944 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.316043 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:46.318467 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.319274 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.321572 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.322016 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.322161 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.326080 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:46.326086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:46.326089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:46.326127 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:46.329219 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.330207 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.330212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.330215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.330348 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.330796 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.330804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.330808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.331195 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:46.331355 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.332104 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.332349 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:10:46.333817 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.311594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:50731)
0927 15:10:46.312536 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:46.312599 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:46.315461 (+ 2862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:46.316367 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:46.316382 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:46.317274 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:46.318229 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.318260 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.319459 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.319467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.321383 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.321400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.322275 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.322282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.322546 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.323345 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:46.323368 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:46.325785 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:46.329442 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:46.329452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:46.329474 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:46.330058 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:46.330475 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:46.330481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:46.330485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:46.330676 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:46.331326 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:46.331334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:46.331713 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 15:10:46.332014 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 15:10:46.332608 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":321,"threads_started":1}
I20260927 15:10:46.335520 16775 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40869
I20260927 15:10:46.335865 16775 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:46.336627 16775 heartbeater.cc:511] Master 127.0.97.62:40869 requested a full tablet report, sending...
I20260927 15:10:46.338312 16458 ts_manager.cc:194] Registered new tserver with Master: fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693)
I20260927 15:10:46.339517 16458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50731
W20260927 15:10:46.667457 16778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:46.667790 16778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:46.667901 16778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:46.675752 16778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:46.675896 16778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:46.676013 16778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:10:46.686311 16778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46571
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:10:46.687722 16778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:46.689015 16778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:46.702941 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:46.703009 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:46.703189 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:46.703933 16778 server_base.cc:1034] running on GCE node
I20260927 15:10:46.704617 16778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:46.705662 16778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:46.706970 16778 hybrid_clock.cc:648] HybridClock initialized: now 1790521846706908 us; error 54 us; skew 500 ppm
I20260927 15:10:46.707418 16778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:10:46 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:46.710952 16778 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:10:46.712743 16778 webserver.cc:460] Webserver started at http://127.0.97.3:46571/ using document root <none> and password file <none>
I20260927 15:10:46.713513 16778 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:46.713692 16778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:46.719954 16778 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:10:46.723272 16793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:46.725013 16778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:10:46.725255 16778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "98982e61e7534fd98736e493fde46c3c"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "34ec7cd6cece09e0611b31185dd81f82"
server_key_iv: "3729f7e56f8c22db75f93e19ecd30d1b"
server_key_version: "encryptionkey@0"
I20260927 15:10:46.726557 16778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:46.750676 16778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:46.755690 16778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:46.755990 16778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:46.757078 16778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:46.758512 16778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:46.758646 16778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:46.758788 16778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:46.758878 16778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:46.794631 16778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38881
I20260927 15:10:46.794672 16905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38881 every 8 connection(s)
I20260927 15:10:46.796277 16778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:46.801277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16778
Sep 27 15:10:46 dist-test-slave-g0w1 krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:10:46.821038 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.799252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42297 (local address 127.0.97.62:40869)
0927 15:10:46.799479 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.799487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.800737 (+ 1250us) server_negotiation.cc:407] Connection header received
0927 15:10:46.803285 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.803291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.803383 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.803496 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:46.805063 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.805835 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.807505 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.807953 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.808109 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.812290 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:46.812295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:46.812298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:46.812346 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:46.817090 (+ 4744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.818091 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.818097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.818099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.818236 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.818811 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.818815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.818817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.819287 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:46.819417 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.820404 (+ 987us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.820672 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:10:46.821753 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.802306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.97.62:40869)
0927 15:10:46.802878 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.802886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.803544 (+ 658us) server_negotiation.cc:407] Connection header received
0927 15:10:46.803762 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.803773 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.803831 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.803957 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:46.805618 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.806456 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.808025 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.808543 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.808690 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.812124 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:46.812130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:46.812134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:46.812172 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:46.817123 (+ 4951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.818920 (+ 1797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.818927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.818933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.819141 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.819601 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.819605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.819607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.819851 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:46.820083 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.821101 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.821322 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":424,"thread_start_us":278,"threads_started":1}
I20260927 15:10:46.822366 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.799808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:42297)
0927 15:10:46.800533 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:46.800577 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:46.802943 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:46.803766 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:46.803776 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:46.804300 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:46.804885 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.804902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.806000 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.806005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.807362 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.807374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.808214 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.808222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.808478 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.809238 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:46.809278 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:46.812036 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:46.817335 (+ 5299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:46.817348 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:46.817369 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:46.817911 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:46.818371 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:46.818376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:46.818379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:46.818680 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:46.819427 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:46.819437 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:46.819887 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 15:10:46.820265 (+ 378us) client_negotiation.cc:240] Negotiation successful
0927 15:10:46.820993 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":291,"threads_started":1}
I20260927 15:10:46.824039 16906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40869
I20260927 15:10:46.824378 16906 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:46.825193 16906 heartbeater.cc:511] Master 127.0.97.62:40869 requested a full tablet report, sending...
I20260927 15:10:46.826941 16458 ts_manager.cc:194] Registered new tserver with Master: 98982e61e7534fd98736e493fde46c3c (127.0.97.3:38881)
I20260927 15:10:46.828037 16458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42297
I20260927 15:10:46.829905 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:46.836768 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.831115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.0.97.62:40869)
0927 15:10:46.831321 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.831328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.831835 (+ 507us) server_negotiation.cc:407] Connection header received
0927 15:10:46.832002 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.832006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.832140 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.832263 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:46.832683 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.833374 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.834856 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.835454 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.835555 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.836022 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:46.836246 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:10:46.836530 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37504: token expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:10:46.836876 16909 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37504: token expired
W20260927 15:10:46.837563 16379 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:40869)
I20260927 15:10:46.851320 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.838053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.0.97.62:40869)
0927 15:10:46.838222 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.838226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.838288 (+ 62us) server_negotiation.cc:407] Connection header received
0927 15:10:46.838408 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.838412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.838468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.838552 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:46.839931 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.840654 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.841990 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.842383 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.842475 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.845228 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:46.845233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:46.845237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:46.845274 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:46.848107 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.848902 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.848906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.848908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.849047 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:46.849500 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:46.849508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:46.849513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:46.849893 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:46.850223 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.850560 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.850813 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:10:46.855126 16458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:10:46.863227 16458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:46.863875 16643 heartbeater.cc:503] Master 127.0.97.62:40869 was elected leader, sending a full tablet report...
I20260927 15:10:46.904469 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.877293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.97.1:40059)
0927 15:10:46.892759 (+ 15466us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.892774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.895276 (+ 2502us) server_negotiation.cc:407] Connection header received
0927 15:10:46.895857 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.895867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.896162 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.896351 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:46.897429 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.898848 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.901823 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.903154 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.903285 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.903534 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.903894 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.904125 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15200,"thread_start_us":15503,"threads_started":1}
I20260927 15:10:46.905488 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.878198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.97.3:38881)
0927 15:10:46.893652 (+ 15454us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.893666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.896234 (+ 2568us) server_negotiation.cc:407] Connection header received
0927 15:10:46.896535 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.896551 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.896890 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.897086 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:46.898318 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.899761 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.902412 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.903890 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.904010 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.904233 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.904899 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.905152 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15228,"thread_start_us":15042,"threads_started":1}
I20260927 15:10:46.905560 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.877405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38881 (local address 127.0.0.1:34750)
0927 15:10:46.896084 (+ 18679us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:46.896112 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:46.896341 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:46.897525 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:46.897533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:46.897569 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:46.898116 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.898130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.899942 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.899950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.902229 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.902249 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.904380 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.904385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.904609 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.904641 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:10:46.904728 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:10:46.905066 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18501,"mutex_wait_us":51,"thread_start_us":18315,"threads_started":1}
I20260927 15:10:46.905598 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.877271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40059 (local address 127.0.0.1:39670)
0927 15:10:46.895000 (+ 17729us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:46.895064 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:46.895526 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:46.896715 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:46.896723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:46.896753 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:46.897206 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.897220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.899029 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.899035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.901669 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.901681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.903381 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.903390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.903601 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.903660 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 15:10:46.904954 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0927 15:10:46.905141 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17361,"mutex_wait_us":49,"thread_start_us":20510,"threads_started":1}
I20260927 15:10:46.906503 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.877874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37693 (local address 127.0.0.1:56968)
0927 15:10:46.898149 (+ 20275us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:46.898172 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:46.898286 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:46.898959 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:46.898963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:46.898992 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:46.899473 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.899514 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.901377 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.901383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.903285 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:46.903302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.905735 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.905741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:46.905921 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.905953 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:10:46.906040 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:10:46.906161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20128,"mutex_wait_us":49,"thread_start_us":16590,"threads_started":1}
I20260927 15:10:46.906651 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:46.878512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.0.97.2:37693)
0927 15:10:46.894204 (+ 15692us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:46.894215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:46.898355 (+ 4140us) server_negotiation.cc:407] Connection header received
0927 15:10:46.898471 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:46.898481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:46.898735 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:46.898934 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:46.899709 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.901229 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.903739 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:46.905222 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:46.905356 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:46.905553 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:46.906135 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 15:10:46.906326 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15490,"thread_start_us":15378,"threads_started":1}
I20260927 15:10:46.909683 16578 tablet_service.cc:1467] Processing CreateTablet for tablet d365936503a94583b48b709dc67fab4b (DEFAULT_TABLE table=test-table [id=dddae3b4f6cc4325a602bdd0a09a1a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:46.909970 16841 tablet_service.cc:1467] Processing CreateTablet for tablet d365936503a94583b48b709dc67fab4b (DEFAULT_TABLE table=test-table [id=dddae3b4f6cc4325a602bdd0a09a1a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:46.911058 16578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d365936503a94583b48b709dc67fab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:46.910588 16710 tablet_service.cc:1467] Processing CreateTablet for tablet d365936503a94583b48b709dc67fab4b (DEFAULT_TABLE table=test-table [id=dddae3b4f6cc4325a602bdd0a09a1a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:46.911516 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d365936503a94583b48b709dc67fab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:46.912014 16710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d365936503a94583b48b709dc67fab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:46.923707 16922 tablet_bootstrap.cc:492] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Bootstrap starting.
I20260927 15:10:46.927388 16923 tablet_bootstrap.cc:492] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Bootstrap starting.
I20260927 15:10:46.929154 16922 tablet_bootstrap.cc:654] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:46.931370 16923 tablet_bootstrap.cc:654] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:46.931815 16922 log.cc:824] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:46.932701 16923 log.cc:824] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:46.933228 16924 tablet_bootstrap.cc:492] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Bootstrap starting.
I20260927 15:10:46.936144 16923 tablet_bootstrap.cc:492] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: No bootstrap required, opened a new log
I20260927 15:10:46.936144 16922 tablet_bootstrap.cc:492] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: No bootstrap required, opened a new log
I20260927 15:10:46.936439 16923 ts_tablet_manager.cc:1397] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 15:10:46.936450 16922 ts_tablet_manager.cc:1397] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260927 15:10:46.937474 16924 tablet_bootstrap.cc:654] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:46.938505 16924 log.cc:824] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:46.941313 16924 tablet_bootstrap.cc:492] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: No bootstrap required, opened a new log
I20260927 15:10:46.941634 16924 ts_tablet_manager.cc:1397] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 15:10:46.947580 16922 raft_consensus.cc:348] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:46.948099 16922 raft_consensus.cc:374] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:46.947875 16923 raft_consensus.cc:348] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:46.948236 16922 raft_consensus.cc:729] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2b562def804a969478a5da77a2fa0b, State: Initialized, Role: FOLLOWER
I20260927 15:10:46.948312 16923 raft_consensus.cc:374] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:46.948410 16923 raft_consensus.cc:729] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98982e61e7534fd98736e493fde46c3c, State: Initialized, Role: FOLLOWER
I20260927 15:10:46.949015 16923 consensus_queue.cc:260] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:46.949004 16922 consensus_queue.cc:260] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:46.950455 16906 heartbeater.cc:503] Master 127.0.97.62:40869 was elected leader, sending a full tablet report...
I20260927 15:10:46.951000 16922 ts_tablet_manager.cc:1428] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 15:10:46.951020 16923 ts_tablet_manager.cc:1428] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 15:10:46.949961 16924 raft_consensus.cc:348] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:46.951588 16924 raft_consensus.cc:374] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:46.951692 16924 raft_consensus.cc:729] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed23c9f59f2401a990718e31f3fc4fe, State: Initialized, Role: FOLLOWER
I20260927 15:10:46.952306 16924 consensus_queue.cc:260] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:46.954142 16775 heartbeater.cc:503] Master 127.0.97.62:40869 was elected leader, sending a full tablet report...
I20260927 15:10:46.954886 16924 ts_tablet_manager.cc:1428] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Time spent starting tablet: real 0.013s user 0.002s sys 0.009s
I20260927 15:10:47.020623 16928 raft_consensus.cc:482] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:47.021063 16928 raft_consensus.cc:504] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:47.022959 16928 leader_election.cc:288] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98982e61e7534fd98736e493fde46c3c (127.0.97.3:38881), fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693)
I20260927 15:10:47.031934 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.023657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35027 (local address 127.0.97.2:37693)
0927 15:10:47.023859 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:47.023866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:47.024581 (+ 715us) server_negotiation.cc:407] Connection header received
0927 15:10:47.024867 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:47.024871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:47.025030 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:47.025159 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:47.025834 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.026677 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.029591 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.030880 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.031037 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.031173 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:47.031508 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:10:47.031657 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:10:47.031934 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.023546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57923 (local address 127.0.97.3:38881)
0927 15:10:47.023783 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:47.023792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:47.024540 (+ 748us) server_negotiation.cc:407] Connection header received
0927 15:10:47.024773 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:47.024781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:47.024864 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:47.024940 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:47.025644 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.026475 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.029337 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.030624 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.030805 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.030935 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:47.031508 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 15:10:47.031675 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:10:47.032232 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.023404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37693 (local address 127.0.97.1:35027)
0927 15:10:47.024432 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:47.024447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:47.024686 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:47.025190 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:47.025193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:47.025232 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:47.025656 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:47.025667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.026858 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.026863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:47.029386 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:47.029401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.031149 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.031156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:47.031314 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.031340 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:10:47.031453 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:10:47.031594 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":40,"spinlock_wait_cycles":43008,"thread_start_us":522,"threads_started":1}
I20260927 15:10:47.032238 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.023309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38881 (local address 127.0.97.1:57923)
0927 15:10:47.024407 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:47.024427 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:47.024607 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:47.024998 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:47.025003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:47.025029 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:47.025477 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:47.025489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.026633 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.026641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:47.029128 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:47.029151 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.031015 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.031022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:47.031263 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.031303 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:10:47.031416 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:10:47.031596 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":52,"thread_start_us":739,"threads_started":1}
I20260927 15:10:47.033705 16861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d365936503a94583b48b709dc67fab4b" candidate_uuid: "8b2b562def804a969478a5da77a2fa0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98982e61e7534fd98736e493fde46c3c" is_pre_election: true
I20260927 15:10:47.033710 16730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d365936503a94583b48b709dc67fab4b" candidate_uuid: "8b2b562def804a969478a5da77a2fa0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed23c9f59f2401a990718e31f3fc4fe" is_pre_election: true
I20260927 15:10:47.034348 16730 raft_consensus.cc:2454] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2b562def804a969478a5da77a2fa0b in term 0.
I20260927 15:10:47.034351 16861 raft_consensus.cc:2454] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2b562def804a969478a5da77a2fa0b in term 0.
I20260927 15:10:47.035210 16533 leader_election.cc:302] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2b562def804a969478a5da77a2fa0b, fed23c9f59f2401a990718e31f3fc4fe; no voters:
I20260927 15:10:47.035704 16928 raft_consensus.cc:2781] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:47.035876 16928 raft_consensus.cc:482] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:47.036031 16928 raft_consensus.cc:3037] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:47.038352 16928 raft_consensus.cc:504] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
I20260927 15:10:47.039179 16928 leader_election.cc:288] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [CANDIDATE]: Term 1 election: Requested vote from peers 98982e61e7534fd98736e493fde46c3c (127.0.97.3:38881), fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693)
I20260927 15:10:47.039790 16730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d365936503a94583b48b709dc67fab4b" candidate_uuid: "8b2b562def804a969478a5da77a2fa0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed23c9f59f2401a990718e31f3fc4fe"
I20260927 15:10:47.039801 16861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d365936503a94583b48b709dc67fab4b" candidate_uuid: "8b2b562def804a969478a5da77a2fa0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98982e61e7534fd98736e493fde46c3c"
I20260927 15:10:47.040027 16730 raft_consensus.cc:3037] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:47.040069 16861 raft_consensus.cc:3037] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:47.042443 16730 raft_consensus.cc:2454] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2b562def804a969478a5da77a2fa0b in term 1.
I20260927 15:10:47.042497 16861 raft_consensus.cc:2454] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2b562def804a969478a5da77a2fa0b in term 1.
I20260927 15:10:47.043128 16531 leader_election.cc:302] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b2b562def804a969478a5da77a2fa0b, 98982e61e7534fd98736e493fde46c3c, fed23c9f59f2401a990718e31f3fc4fe; no voters:
I20260927 15:10:47.043565 16928 raft_consensus.cc:2781] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:47.044127 16928 raft_consensus.cc:686] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 1 LEADER]: Becoming Leader. State: Replica: 8b2b562def804a969478a5da77a2fa0b, State: Running, Role: LEADER
I20260927 15:10:47.044741 16928 consensus_queue.cc:237] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } }
W20260927 15:10:47.048090 16907 tablet.cc:2369] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:47.050153 16458 catalog_manager.cc:5495] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2b562def804a969478a5da77a2fa0b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8b2b562def804a969478a5da77a2fa0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2b562def804a969478a5da77a2fa0b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 } health_report { overall_health: UNKNOWN } } }
W20260927 15:10:47.060639 16776 tablet.cc:2369] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:47.073587 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.067623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.0.97.1:40059)
0927 15:10:47.067871 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:47.067880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:47.067903 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:10:47.068247 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:47.068251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:47.068348 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:47.068546 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:47.068929 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.069684 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.071229 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.071729 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.071873 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.072429 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:47.072973 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:47.073147 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:47.073205 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 15:10:47.073295 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 15:10:47.075325 16644 tablet.cc:2369] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:47.082748 16730 raft_consensus.cc:1261] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Refusing update from remote peer 8b2b562def804a969478a5da77a2fa0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:47.082760 16861 raft_consensus.cc:1261] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Refusing update from remote peer 8b2b562def804a969478a5da77a2fa0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:47.084105 16933 consensus_queue.cc:1035] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "98982e61e7534fd98736e493fde46c3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:47.084512 16928 consensus_queue.cc:1035] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed23c9f59f2401a990718e31f3fc4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:47.095938 16936 mvcc.cc:205] Tried to move back new op lower bound from 7333977485640544256 to 7333977485499629568. Current Snapshot: MvccSnapshot[applied={T|T < 7333977485640544256}]
I20260927 15:10:47.113284 16458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:47.113600 16458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:47.118247 16458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Sending DeleteTablet for 3 replicas of tablet d365936503a94583b48b709dc67fab4b
I20260927 15:10:47.119531 16578 tablet_service.cc:1514] Processing DeleteTablet for tablet d365936503a94583b48b709dc67fab4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39670
I20260927 15:10:47.119580 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet d365936503a94583b48b709dc67fab4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34750
I20260927 15:10:47.119558 16710 tablet_service.cc:1514] Processing DeleteTablet for tablet d365936503a94583b48b709dc67fab4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56968
I20260927 15:10:47.120481 16946 tablet_replica.cc:330] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: stopping tablet replica
I20260927 15:10:47.120571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 16495
I20260927 15:10:47.120940 16946 raft_consensus.cc:2229] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:47.121634 16946 raft_consensus.cc:2258] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:47.124194 16946 ts_tablet_manager.cc:1905] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:47.132912 16946 ts_tablet_manager.cc:1918] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:47.133184 16947 tablet_replica.cc:330] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: stopping tablet replica
I20260927 15:10:47.133217 16946 log.cc:1196] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/d365936503a94583b48b709dc67fab4b
I20260927 15:10:47.133790 16948 tablet_replica.cc:330] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: stopping tablet replica
I20260927 15:10:47.133876 16946 ts_tablet_manager.cc:1939] T d365936503a94583b48b709dc67fab4b P 8b2b562def804a969478a5da77a2fa0b: Deleting consensus metadata
I20260927 15:10:47.134424 16948 raft_consensus.cc:2229] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:47.134995 16948 raft_consensus.cc:2258] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:47.135528 16947 raft_consensus.cc:2229] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:47.136086 16947 raft_consensus.cc:2258] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:47.136183 16443 catalog_manager.cc:4841] TS 8b2b562def804a969478a5da77a2fa0b (127.0.97.1:40059): tablet d365936503a94583b48b709dc67fab4b (table test-table [id=dddae3b4f6cc4325a602bdd0a09a1a42]) successfully deleted
I20260927 15:10:47.138689 16947 ts_tablet_manager.cc:1905] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:47.141556 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.122261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.97.1:40059)
0927 15:10:47.122436 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:47.122442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:47.122460 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:10:47.122534 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:47.122542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:47.122656 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:47.122761 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:47.125050 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.126214 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.128293 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.129012 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.129141 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.133124 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:47.133141 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:47.133165 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:47.133234 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:47.138180 (+ 4946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:47.139156 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:47.139165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:47.139168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:47.139353 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:47.139806 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:47.139811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:47.139815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:47.140189 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:47.140519 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:47.140863 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:10:47.141169 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:10:47.142390 16638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:47.145226 16948 ts_tablet_manager.cc:1905] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:47.150081 16947 ts_tablet_manager.cc:1918] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:47.150382 16947 log.cc:1196] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/d365936503a94583b48b709dc67fab4b
I20260927 15:10:47.151115 16947 ts_tablet_manager.cc:1939] T d365936503a94583b48b709dc67fab4b P 98982e61e7534fd98736e493fde46c3c: Deleting consensus metadata
I20260927 15:10:47.151554 16948 ts_tablet_manager.cc:1918] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:47.151769 16948 log.cc:1196] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/d365936503a94583b48b709dc67fab4b
I20260927 15:10:47.152320 16948 ts_tablet_manager.cc:1939] T d365936503a94583b48b709dc67fab4b P fed23c9f59f2401a990718e31f3fc4fe: Deleting consensus metadata
I20260927 15:10:47.153352 16442 catalog_manager.cc:4841] TS 98982e61e7534fd98736e493fde46c3c (127.0.97.3:38881): tablet d365936503a94583b48b709dc67fab4b (table test-table [id=dddae3b4f6cc4325a602bdd0a09a1a42]) successfully deleted
I20260927 15:10:47.154138 16444 catalog_manager.cc:4841] TS fed23c9f59f2401a990718e31f3fc4fe (127.0.97.2:37693): tablet d365936503a94583b48b709dc67fab4b (table test-table [id=dddae3b4f6cc4325a602bdd0a09a1a42]) successfully deleted
I20260927 15:10:47.244151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 16495
I20260927 15:10:47.264317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 16647
I20260927 15:10:47.284312 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.265219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.0.97.2:37693)
0927 15:10:47.265441 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:47.265447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:47.265471 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:10:47.265630 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:47.265638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:47.265712 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:47.265852 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:47.267394 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.268177 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.269631 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.270059 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.270212 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.277631 (+ 7419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:47.277649 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:47.277666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:47.277707 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:47.281053 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:47.281994 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:47.282003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:47.282006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:47.282152 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:47.282589 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:47.282595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:47.282600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:47.283002 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:47.283387 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:47.283703 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:10:47.283941 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:10:47.285151 16770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:47.379472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 16647
I20260927 15:10:47.399266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 16778
I20260927 15:10:47.415676 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:47.400685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.0.97.3:38881)
0927 15:10:47.400896 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:47.400902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:47.400929 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:10:47.401003 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:47.401008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:47.401085 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:47.401199 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:47.402596 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.403432 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.404883 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:47.405310 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:47.405459 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:47.408611 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:47.408628 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:47.408645 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:47.408689 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:47.412157 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:47.413137 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:47.413145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:47.413148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:47.413348 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:47.413824 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:47.413831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:47.413842 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:47.414269 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:47.414631 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:47.414961 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:10:47.415284 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:10:47.416548 16901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:47.508510 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 16778
I20260927 15:10:47.525743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 16425
I20260927 15:10:47.527027 16488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:10:47.644479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 16425
I20260927 15:11:07.667395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40869
--webserver_interface=127.0.97.62
--webserver_port=38353
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:08.031363 16953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:08.031692 16953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:08.031850 16953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:08.039584 16953 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:11:08.039706 16953 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:11:08.039744 16953 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:11:08.039834 16953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:08.039919 16953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:08.040019 16953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:08.040107 16953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:08.040195 16953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:08.050788 16953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:08.052316 16953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:08.053774 16953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:08.066494 16958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:08.066546 16959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:08.066573 16961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:08.068116 16953 server_base.cc:1034] running on GCE node
I20260927 15:11:08.069103 16953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:08.070879 16953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:08.072170 16953 hybrid_clock.cc:648] HybridClock initialized: now 1790521868072080 us; error 88 us; skew 500 ppm
I20260927 15:11:08.072584 16953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:08 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:08.083518 16953 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:11:08.085357 16953 webserver.cc:460] Webserver started at http://127.0.97.62:38353/ using document root <none> and password file <none>
I20260927 15:11:08.086206 16953 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:08.086362 16953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:08.092861 16953 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:11:08.096221 16968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:08.097803 16953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:08.098060 16953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "a157ac72fdc347e3b2800b09d3756c5c"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "64b27b6b8e907c4671e1a5a513c02c4d"
server_key_iv: "290f349ed70760c7374ab0ea9336710a"
server_key_version: "encryptionkey@0"
I20260927 15:11:08.099143 16953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:08.115214 16953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:08.120326 16953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:08.120682 16953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:08.141917 16953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40869
I20260927 15:11:08.141924 17019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40869 every 8 connection(s)
I20260927 15:11:08.143709 16953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:08.144932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 16953
I20260927 15:11:08.146270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40059
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40869
--webserver_port=45317
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:08.153158 17020 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:11:08.156841 17020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap starting.
I20260927 15:11:08.178884 17020 log.cc:824] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:08.195943 17020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:11:08.196595 17020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Bootstrap complete.
I20260927 15:11:08.206977 17020 raft_consensus.cc:348] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:11:08.208091 17020 raft_consensus.cc:729] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157ac72fdc347e3b2800b09d3756c5c, State: Initialized, Role: FOLLOWER
I20260927 15:11:08.208860 17020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:11:08.209117 17020 raft_consensus.cc:388] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:08.209247 17020 raft_consensus.cc:482] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:08.209458 17020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:11:08.213725 17020 raft_consensus.cc:504] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:11:08.214319 17020 leader_election.cc:302] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a157ac72fdc347e3b2800b09d3756c5c; no voters:
I20260927 15:11:08.215144 17020 leader_election.cc:288] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:11:08.215276 17024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:11:08.216451 17024 raft_consensus.cc:686] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [term 3 LEADER]: Becoming Leader. State: Replica: a157ac72fdc347e3b2800b09d3756c5c, State: Running, Role: LEADER
I20260927 15:11:08.217145 17024 consensus_queue.cc:237] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } }
I20260927 15:11:08.217614 17020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:08.219822 17026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a157ac72fdc347e3b2800b09d3756c5c. Latest consensus state: current_term: 3 leader_uuid: "a157ac72fdc347e3b2800b09d3756c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } } }
I20260927 15:11:08.220259 17026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:08.220889 17025 sys_catalog.cc:455] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a157ac72fdc347e3b2800b09d3756c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a157ac72fdc347e3b2800b09d3756c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40869 } } }
I20260927 15:11:08.221204 17025 sys_catalog.cc:458] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:08.230656 17031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:08.242066 17031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:08.243850 17031 catalog_manager.cc:1259] Loaded cluster ID: 4d217da13bfe4a21a8eaa20f44d1afee
I20260927 15:11:08.244160 17031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:08.248850 17031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:08.250934 17031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Loaded TSK: 0
I20260927 15:11:08.251227 17031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a157ac72fdc347e3b2800b09d3756c5c: Loaded TSK: 1
F20260927 15:11:08.252538 17031 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3a662189ad google::LogMessage::Fail() at ??:0
@ 0x7f3a6621b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3a662184de google::LogMessage::Flush() at ??:0
@ 0x7f3a66219a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3a69f7c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3a69f6b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3a7bc4b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3a7bc4a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3a7bcd62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3a7bcd60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3a7bd205e9 std::function<>::operator()() at ??:0
@ 0x7f3a7bc4ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3a7bc4cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3a7bcd4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3a7bcd4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3a68e64345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3a68e6791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3a68e67799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3a68e42eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3a6c11c6db start_thread at ??:0
@ 0x7f3a650b261f clone at ??:0
W20260927 15:11:08.558920 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:08.559367 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:08.559548 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:08.567458 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:08.567636 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:08.567811 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:08.578780 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45317
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:08.580255 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:08.581696 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:08.599689 17068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:08.599664 17067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:08.599790 17070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:08.600452 17022 server_base.cc:1034] running on GCE node
I20260927 15:11:08.601269 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:08.602370 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:08.603626 17022 hybrid_clock.cc:648] HybridClock initialized: now 1790521868603576 us; error 41 us; skew 500 ppm
I20260927 15:11:08.604004 17022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:08 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:08.607618 17022 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:11:08.609349 17022 webserver.cc:460] Webserver started at http://127.0.97.1:45317/ using document root <none> and password file <none>
I20260927 15:11:08.610195 17022 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:08.610380 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:08.616716 17022 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:08.620149 17077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:08.622164 17022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:08.622395 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "8b2b562def804a969478a5da77a2fa0b"
format_stamp: "Formatted at 2026-09-27 15:10:22 on dist-test-slave-g0w1"
server_key: "f5e859e88501cc3b78a01030c5298c34"
server_key_iv: "a279363403fde5fa3277490ad270e94a"
server_key_version: "encryptionkey@0"
I20260927 15:11:08.623664 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:08.638547 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:08.643692 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:08.644016 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:08.645071 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:08.646608 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:08.646740 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:08.646883 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:08.646975 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:08.683568 17022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40059
I20260927 15:11:08.683662 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40059 every 8 connection(s)
I20260927 15:11:08.685153 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:08.690555 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:08.688609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:35337)
0927 15:11:08.689414 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:08.689619 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
W20260927 15:11:08.691311 17190 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40869 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40869: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
I20260927 15:11:08.692214 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:08.691872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:41859)
0927 15:11:08.692027 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:08.692105 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:11:08.693007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17022
I20260927 15:11:08.693069 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:08.692734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:44109)
0927 15:11:08.692872 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:08.692957 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:11:08.693375 17190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:11:08.693687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37693
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40869
--webserver_port=35503
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:09.042896 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:09.043334 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:09.043515 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:09.051239 17193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:09.051367 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:09.051564 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:09.061691 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35503
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:09.063134 17193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:09.064548 17193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:09.078974 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:09.079105 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:09.079097 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:09.080014 17193 server_base.cc:1034] running on GCE node
I20260927 15:11:09.080766 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:09.081879 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:09.083190 17193 hybrid_clock.cc:648] HybridClock initialized: now 1790521869083126 us; error 50 us; skew 500 ppm
I20260927 15:11:09.083554 17193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:09 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:09.086967 17193 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:11:09.088827 17193 webserver.cc:460] Webserver started at http://127.0.97.2:35503/ using document root <none> and password file <none>
I20260927 15:11:09.089573 17193 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:09.089679 17193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:09.096058 17193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:09.099396 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:09.101100 17193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:09.101265 17193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "fed23c9f59f2401a990718e31f3fc4fe"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "35955ddd3e1b15a422876894469729e1"
server_key_iv: "a09aab771edd42c80ddcc83a8523e192"
server_key_version: "encryptionkey@0"
I20260927 15:11:09.102317 17193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:09.126528 17193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:09.131453 17193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:09.131752 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:09.132831 17193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:09.134239 17193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:09.134353 17193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:09.134481 17193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:09.134577 17193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:09.169842 17193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37693
I20260927 15:11:09.169960 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37693 every 8 connection(s)
I20260927 15:11:09.171468 17193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:09.176515 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.174575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:54753)
0927 15:11:09.175376 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.175597 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":285,"threads_started":1}
W20260927 15:11:09.177477 17321 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40869 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40869: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
I20260927 15:11:09.178371 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.178043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:56167)
0927 15:11:09.178189 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.178261 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:11:09.179446 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.179145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:54743)
0927 15:11:09.179270 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.179343 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:11:09.179698 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17193
W20260927 15:11:09.179880 17321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:11:09.180374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38881
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40869
--webserver_port=46571
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:09.529696 17324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:09.530021 17324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:09.530125 17324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:09.537614 17324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:09.537747 17324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:09.537864 17324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:09.547942 17324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46571
--tserver_master_addrs=127.0.97.62:40869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:09.549311 17324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:09.550671 17324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:09.565122 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:09.565151 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:09.565294 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:09.566094 17324 server_base.cc:1034] running on GCE node
I20260927 15:11:09.566738 17324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:09.567838 17324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:09.569073 17324 hybrid_clock.cc:648] HybridClock initialized: now 1790521869569005 us; error 57 us; skew 500 ppm
I20260927 15:11:09.569454 17324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:09 dist-test-slave-g0w1 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:09.573267 17324 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:11:09.575026 17324 webserver.cc:460] Webserver started at http://127.0.97.3:46571/ using document root <none> and password file <none>
I20260927 15:11:09.575877 17324 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:09.575994 17324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:09.582201 17324 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:09.585556 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:09.587509 17324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:09.587739 17324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "98982e61e7534fd98736e493fde46c3c"
format_stamp: "Formatted at 2026-09-27 15:10:23 on dist-test-slave-g0w1"
server_key: "34ec7cd6cece09e0611b31185dd81f82"
server_key_iv: "3729f7e56f8c22db75f93e19ecd30d1b"
server_key_version: "encryptionkey@0"
I20260927 15:11:09.588955 17324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:09.614614 17324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:09.620082 17324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:09.620417 17324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:09.621569 17324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:09.623257 17324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:09.623405 17324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:09.623561 17324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:09.623656 17324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:09.662392 17324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38881
I20260927 15:11:09.662489 17451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38881 every 8 connection(s)
I20260927 15:11:09.664022 17324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:09.665908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17324
I20260927 15:11:09.669727 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.667699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:36321)
0927 15:11:09.668512 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.668729 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":301,"threads_started":1}
W20260927 15:11:09.670488 17452 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40869 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40869: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
I20260927 15:11:09.671665 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.671256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:39049)
0927 15:11:09.671439 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.671525 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:11:09.672724 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.672307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:46951)
0927 15:11:09.672507 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.672597 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260927 15:11:09.673012 17452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:11:09.694983 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.694024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:48079)
0927 15:11:09.694639 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.694738 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":331,"threads_started":1}
I20260927 15:11:10.181341 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:10.180539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:60689)
0927 15:11:10.181007 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:10.181097 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260927 15:11:10.674558 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:10.673626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:36035)
0927 15:11:10.674190 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:10.674284 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":297,"threads_started":1}
I20260927 15:11:10.696928 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:10.695996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:44913)
0927 15:11:10.696657 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:10.696756 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":412,"threads_started":1}
I20260927 15:11:11.183269 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:11.182361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:47535)
0927 15:11:11.182883 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:11.182980 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260927 15:11:11.676510 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:11.675682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:36099)
0927 15:11:11.676233 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:11.676323 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":307,"threads_started":1}
I20260927 15:11:11.698768 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:11.697925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:48939)
0927 15:11:11.698446 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:11.698551 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260927 15:11:12.185369 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:12.184472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:36309)
0927 15:11:12.185069 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:12.185170 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260927 15:11:12.678313 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:12.677534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:53225)
0927 15:11:12.678020 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:12.678106 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260927 15:11:12.700698 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:12.699898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:42257)
0927 15:11:12.700431 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:12.700521 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":240,"threads_started":1}
I20260927 15:11:13.187336 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:13.186386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:35995)
0927 15:11:13.186998 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:13.187135 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":290,"threads_started":1}
I20260927 15:11:13.680164 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:13.679433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:47867)
0927 15:11:13.679902 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:13.679987 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260927 15:11:13.702612 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:13.701804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:48721)
0927 15:11:13.702340 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:13.702430 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":268,"threads_started":1}
I20260927 15:11:14.189287 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:14.188431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:51773)
0927 15:11:14.189002 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:14.189094 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":282,"threads_started":1}
I20260927 15:11:14.682096 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:14.681322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:57051)
0927 15:11:14.681809 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:14.681909 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":255,"threads_started":1}
I20260927 15:11:14.704457 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:14.703710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:58227)
0927 15:11:14.704201 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:14.704290 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
I20260927 15:11:15.191350 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:15.190433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:59389)
0927 15:11:15.190957 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:15.191177 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260927 15:11:15.684137 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:15.683237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:59347)
0927 15:11:15.683785 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:15.683943 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
I20260927 15:11:15.706485 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:15.705463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:37097)
0927 15:11:15.706123 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:15.706214 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":299,"threads_started":1}
I20260927 15:11:16.193451 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:16.192441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:58467)
0927 15:11:16.193066 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:16.193229 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":243,"threads_started":1}
I20260927 15:11:16.686092 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:16.685196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:53731)
0927 15:11:16.685724 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:16.685834 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":228,"threads_started":1}
I20260927 15:11:16.708401 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:16.707663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:45993)
0927 15:11:16.708141 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:16.708226 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260927 15:11:17.195441 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:17.194634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:37245)
0927 15:11:17.195167 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:17.195258 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260927 15:11:17.687865 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:17.687178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:54157)
0927 15:11:17.687630 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:17.687707 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260927 15:11:17.710268 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:17.709450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:52485)
0927 15:11:17.709981 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:17.710065 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":240,"threads_started":1}
I20260927 15:11:18.197400 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.196551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:39515)
0927 15:11:18.197095 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.197184 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":292,"threads_started":1}
I20260927 15:11:18.689715 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.688894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:54715)
0927 15:11:18.689452 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.689540 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":289,"threads_started":1}
I20260927 15:11:18.712176 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.711396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:56573)
0927 15:11:18.711901 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.711996 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260927 15:11:19.199285 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.198495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:46929)
0927 15:11:19.198955 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.199046 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260927 15:11:19.691670 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.690864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:57161)
0927 15:11:19.691424 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.691509 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":327,"threads_started":1}
I20260927 15:11:19.714175 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.713268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:34875)
0927 15:11:19.713887 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.713991 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":322,"threads_started":1}
I20260927 15:11:20.201167 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.200337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:35201)
0927 15:11:20.200861 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.200965 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":253,"threads_started":1}
I20260927 15:11:20.693641 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.692760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:51695)
0927 15:11:20.693255 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.693353 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
I20260927 15:11:20.716311 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.715524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:48379)
0927 15:11:20.716055 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.716145 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":269,"threads_started":1}
I20260927 15:11:21.203192 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:21.202285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:48283)
0927 15:11:21.202801 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:21.202943 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":261,"threads_started":1}
I20260927 15:11:21.695586 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:21.694765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:54993)
0927 15:11:21.695293 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:21.695391 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":232,"threads_started":1}
I20260927 15:11:21.718396 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:21.717467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:55729)
0927 15:11:21.718092 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:21.718186 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":315,"threads_started":1}
I20260927 15:11:22.205469 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.204446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:51719)
0927 15:11:22.205106 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.205231 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":282,"threads_started":1}
I20260927 15:11:22.697500 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.696691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:48909)
0927 15:11:22.697189 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.697285 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260927 15:11:22.720225 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.719525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:39031)
0927 15:11:22.719955 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.720038 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260927 15:11:23.207662 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.206676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:47839)
0927 15:11:23.207323 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.207466 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":332,"threads_started":1}
I20260927 15:11:23.699478 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.698623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:35253)
0927 15:11:23.699188 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.699282 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":277,"threads_started":1}
I20260927 15:11:23.722381 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.721601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:39353)
0927 15:11:23.722103 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.722198 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260927 15:11:24.209604 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.208862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.2:38523)
0927 15:11:24.209317 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:24.209412 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260927 15:11:24.667589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17022
I20260927 15:11:24.695742 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.668851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.97.1:40059)
0927 15:11:24.669597 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:24.669620 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:24.669679 (+ 59us) server_negotiation.cc:407] Connection header received
0927 15:11:24.671908 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:24.671920 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:24.672427 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:24.672781 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:24.674326 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.675760 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.677201 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.678137 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.678299 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:24.688717 (+ 10418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:24.688728 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:24.688747 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:24.688793 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:24.692236 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.693009 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.693017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.693025 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.693220 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.693635 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.693639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.693641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.694271 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:24.694581 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:24.695014 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 15:11:24.695449 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":338,"threads_started":1}
I20260927 15:11:24.697188 17185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:24.701723 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.700834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.3:57757)
0927 15:11:24.701344 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:24.701475 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":284,"threads_started":1}
I20260927 15:11:24.784796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17022
I20260927 15:11:24.784896 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.783718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40869 (local address 127.0.97.1:33641)
0927 15:11:24.784323 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:24.784537 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
I20260927 15:11:24.802263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17193
I20260927 15:11:24.821137 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.803333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.97.2:37693)
0927 15:11:24.803794 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:24.803808 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:24.803845 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:11:24.805335 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:24.805345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:24.805856 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:24.806233 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:24.807901 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.809211 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.810490 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.811428 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.811640 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:24.814341 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:24.814349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:24.814366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:24.814417 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:24.817621 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.818373 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.818381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.818388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.818567 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.818953 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.818958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.818970 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.819582 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:24.819909 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:24.820383 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 15:11:24.820833 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":205,"threads_started":1}
I20260927 15:11:24.822598 17316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:24.909415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17193
I20260927 15:11:24.928030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17324
I20260927 15:11:24.947722 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.929012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.97.3:38881)
0927 15:11:24.929572 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:24.929586 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:24.929629 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:11:24.931378 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:24.931389 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:24.931860 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:24.932252 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:24.933744 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.935003 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.936432 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.937165 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.937382 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:24.940647 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:24.940657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:24.940676 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:24.940724 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:24.943864 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.944676 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.944684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.944691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.944883 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.945311 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.945317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.945320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.945915 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:24.946227 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:24.946717 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 15:11:24.947389 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":255,"threads_started":1}
I20260927 15:11:24.949190 17447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:25.035130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17324
2026-09-27T15:11:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64788 ms)
[----------] 30 tests from SecurityITest (307476 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:25 dist-test-slave-g0w1 krb5kdc[17510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:25 dist-test-slave-g0w1 krb5kdc[17510](info): set up 2 sockets
Sep 27 15:11:25 dist-test-slave-g0w1 krb5kdc[17510](info): commencing operation
krb5kdc: starting...
W20260927 15:11:26.167340 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:26 dist-test-slave-g0w1 krb5kdc[17510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:11:26.428501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:26.772944 17523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:26.773342 17523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:26.773524 17523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:26.781467 17523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:26.781639 17523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:26.781747 17523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:26.781843 17523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:26.781935 17523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:26.792359 17523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:26.793783 17523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:26.795233 17523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:26.808563 17528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.808611 17529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.808631 17531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:26.809284 17523 server_base.cc:1034] running on GCE node
I20260927 15:11:26.810204 17523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:26.811820 17523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:26.813107 17523 hybrid_clock.cc:648] HybridClock initialized: now 1790521886813050 us; error 37 us; skew 500 ppm
I20260927 15:11:26.813560 17523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:26 dist-test-slave-g0w1 krb5kdc[17510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:26.817047 17523 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:11:26.818849 17523 webserver.cc:460] Webserver started at http://127.0.97.62:39381/ using document root <none> and password file <none>
I20260927 15:11:26.819705 17523 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:26.819856 17523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:26.820186 17523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:26.823647 17523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "4d65f9f87e0d449f893f9343bcbf2e46"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-g0w1"
server_key: "dd3330bbe44cc81c0e0fc31b2743c694"
server_key_iv: "f7888256bf4e04ce1cbe7c5d476e97a8"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.824496 17523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d65f9f87e0d449f893f9343bcbf2e46"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-g0w1"
server_key: "dd3330bbe44cc81c0e0fc31b2743c694"
server_key_iv: "f7888256bf4e04ce1cbe7c5d476e97a8"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.830374 17523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:11:26.834453 17538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.836278 17523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:26.836557 17523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "4d65f9f87e0d449f893f9343bcbf2e46"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-g0w1"
server_key: "dd3330bbe44cc81c0e0fc31b2743c694"
server_key_iv: "f7888256bf4e04ce1cbe7c5d476e97a8"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.836802 17523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:26.865062 17523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:26.869908 17523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:26.870258 17523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:26.890596 17523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34799
I20260927 15:11:26.890645 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34799 every 8 connection(s)
I20260927 15:11:26.892228 17523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:26.894166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17523
I20260927 15:11:26.894373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:26.895004 388 external_mini_cluster.cc:1428] Setting key f7191a91ce66e2362425e9310d69ecbe
I20260927 15:11:26.896942 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:11:26 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:26.911222 17590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46: Bootstrap starting.
I20260927 15:11:26.915570 17590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:26.916926 17590 log.cc:824] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:26.920773 17590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46: No bootstrap required, opened a new log
I20260927 15:11:26.921804 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.897305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.0.97.62:34799)
0927 15:11:26.898165 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.898178 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.898222 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:11:26.900003 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.900029 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.900474 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.900866 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:26.902938 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.905158 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.906624 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.907770 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.907995 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.912204 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:26.912215 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:26.912236 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:26.912293 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:26.916464 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:26.917330 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:26.917342 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:26.917354 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:26.917617 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:26.918280 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:26.918284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:26.918286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:26.919107 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:26.919400 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.919881 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.920342 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":385,"threads_started":1}
I20260927 15:11:26.929605 17590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260927 15:11:26.930042 17590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:26.930177 17590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d65f9f87e0d449f893f9343bcbf2e46, State: Initialized, Role: FOLLOWER
I20260927 15:11:26.930853 17590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260927 15:11:26.931178 17590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:26.931347 17590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:26.931527 17590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:26.933693 17590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260927 15:11:26.934216 17590 leader_election.cc:302] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d65f9f87e0d449f893f9343bcbf2e46; no voters:
I20260927 15:11:26.934894 17590 leader_election.cc:288] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:26.935040 17595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:26.935729 17595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [term 1 LEADER]: Becoming Leader. State: Replica: 4d65f9f87e0d449f893f9343bcbf2e46, State: Running, Role: LEADER
I20260927 15:11:26.936372 17595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260927 15:11:26.937364 17590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:26.939697 17597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d65f9f87e0d449f893f9343bcbf2e46. Latest consensus state: current_term: 1 leader_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } } }
I20260927 15:11:26.940143 17597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:26.940232 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d65f9f87e0d449f893f9343bcbf2e46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } } }
I20260927 15:11:26.940704 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:26.943427 17603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:26.948355 17603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:26.957530 17603 catalog_manager.cc:1347] Generated new cluster ID: 9eee373285bd4c90aa1b4b69d440a0bc
I20260927 15:11:26.957662 17603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:26.985986 17603 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:26.988183 17603 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:27.012945 17603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46: Generated new TSK 0
I20260927 15:11:27.014113 17603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:11:27.084041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:27.432859 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:27.433149 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:27.433246 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:27.440982 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:27.441129 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:27.441309 17616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:27.452024 17616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:27.453388 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:27.454735 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:27.468684 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:27.468665 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:27.469259 17621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:27.469660 17616 server_base.cc:1034] running on GCE node
I20260927 15:11:27.470260 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:27.471308 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:27.472568 17616 hybrid_clock.cc:648] HybridClock initialized: now 1790521887472499 us; error 50 us; skew 500 ppm
I20260927 15:11:27.472993 17616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:27 dist-test-slave-g0w1 krb5kdc[17510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:27.476760 17616 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:11:27.478413 17616 webserver.cc:460] Webserver started at http://127.0.97.1:43591/ using document root <none> and password file <none>
I20260927 15:11:27.479166 17616 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:27.479270 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:27.479548 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:27.482514 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "691b65e0c85a454f8fd5733a8ffaaa68"
format_stamp: "Formatted at 2026-09-27 15:11:27 on dist-test-slave-g0w1"
server_key: "7e9fc9f0a29041aa77b1550a231f02dd"
server_key_iv: "b60f444441fdbc79cd38f89d6e238aae"
server_key_version: "encryptionkey@0"
I20260927 15:11:27.483356 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "691b65e0c85a454f8fd5733a8ffaaa68"
format_stamp: "Formatted at 2026-09-27 15:11:27 on dist-test-slave-g0w1"
server_key: "7e9fc9f0a29041aa77b1550a231f02dd"
server_key_iv: "b60f444441fdbc79cd38f89d6e238aae"
server_key_version: "encryptionkey@0"
I20260927 15:11:27.488788 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:11:27.492865 17631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:27.494509 17616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:11:27.494720 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "691b65e0c85a454f8fd5733a8ffaaa68"
format_stamp: "Formatted at 2026-09-27 15:11:27 on dist-test-slave-g0w1"
server_key: "7e9fc9f0a29041aa77b1550a231f02dd"
server_key_iv: "b60f444441fdbc79cd38f89d6e238aae"
server_key_version: "encryptionkey@0"
I20260927 15:11:27.494964 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:27.511296 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:27.516346 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:27.516673 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:27.517716 17616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:27.519285 17616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:27.519359 17616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:27.519503 17616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:27.519588 17616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:27.552173 17616 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34167
I20260927 15:11:27.552223 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34167 every 8 connection(s)
I20260927 15:11:27.553897 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:27.559806 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17616
I20260927 15:11:27.560012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:27.560701 388 external_mini_cluster.cc:1428] Setting key 54b5e3da88ba6b805d9b7f20093528f7
Sep 27 15:11:27 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:27.574810 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:27.556566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49161 (local address 127.0.97.62:34799)
0927 15:11:27.557046 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:27.557052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:27.558087 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 15:11:27.560016 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:27.560021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:27.560121 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:27.560239 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:27.561832 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:27.562614 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:27.564243 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:27.564648 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:27.564783 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:27.568541 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:27.568546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:27.568550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:27.568583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:27.571399 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:27.572412 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:27.572416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:27.572418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:27.572545 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:27.573051 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:27.573054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:27.573056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:27.573313 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:27.573421 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:27.574358 (+ 937us) server_negotiation.cc:299] Negotiation successful
0927 15:11:27.574539 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
I20260927 15:11:27.576406 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:27.557192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.1:49161)
0927 15:11:27.557926 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:27.557968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:27.559730 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:27.560491 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:27.560500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:27.561078 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:27.561659 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:27.561677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:27.562783 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:27.562788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:27.564089 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:27.564102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:27.564889 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:27.564898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:27.565140 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:27.565830 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:27.565854 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:27.568294 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:27.571568 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:27.571580 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:27.571601 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:27.572269 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:27.572662 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:27.572670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:27.572677 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:27.572941 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:27.573442 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:27.573454 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:27.573941 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 15:11:27.574256 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 15:11:27.574844 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":301,"threads_started":1}
I20260927 15:11:27.578231 17744 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34799
I20260927 15:11:27.578584 17744 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:27.579407 17744 heartbeater.cc:511] Master 127.0.97.62:34799 requested a full tablet report, sending...
I20260927 15:11:27.581774 17555 ts_manager.cc:194] Registered new tserver with Master: 691b65e0c85a454f8fd5733a8ffaaa68 (127.0.97.1:34167)
I20260927 15:11:27.583355 17555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49161
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:11:27.618681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:27.959121 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:27.959625 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:27.959805 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:27.968175 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:27.968340 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:27.968515 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:27.978607 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:27.980055 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:27.981397 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:27.996313 17755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:27.999173 17758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:27.999202 17750 server_base.cc:1034] running on GCE node
W20260927 15:11:27.999210 17756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:28.000166 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:28.001166 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:28.002439 17750 hybrid_clock.cc:648] HybridClock initialized: now 1790521888002384 us; error 40 us; skew 500 ppm
I20260927 15:11:28.002792 17750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:28 dist-test-slave-g0w1 krb5kdc[17510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:28.006189 17750 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:11:28.007759 17750 webserver.cc:460] Webserver started at http://127.0.97.2:36485/ using document root <none> and password file <none>
I20260927 15:11:28.008466 17750 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:28.008605 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:28.008924 17750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:28.012056 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "241a7c8219624719bd141e1065a84a78"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-g0w1"
server_key: "7a117f067ff55f3d545446a85a3ff64e"
server_key_iv: "2977d30bac87d1b9eb4b2aecf3ad7ce3"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.012877 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "241a7c8219624719bd141e1065a84a78"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-g0w1"
server_key: "7a117f067ff55f3d545446a85a3ff64e"
server_key_iv: "2977d30bac87d1b9eb4b2aecf3ad7ce3"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.018198 17750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:11:28.021970 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.023504 17750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:11:28.023677 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "241a7c8219624719bd141e1065a84a78"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-g0w1"
server_key: "7a117f067ff55f3d545446a85a3ff64e"
server_key_iv: "2977d30bac87d1b9eb4b2aecf3ad7ce3"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.023859 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:28.040534 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:28.044905 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:28.045161 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:28.046164 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:28.047588 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:28.047660 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.047748 17750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:28.047791 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.081730 17750 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45053
I20260927 15:11:28.081758 17877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45053 every 8 connection(s)
I20260927 15:11:28.083138 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:28.084321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17750
I20260927 15:11:28.084506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:28.085104 388 external_mini_cluster.cc:1428] Setting key 503b552c55df75177e7e6c827015dc64
Sep 27 15:11:28 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:28.105909 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.086031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56735 (local address 127.0.97.62:34799)
0927 15:11:28.086482 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.086486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.087839 (+ 1353us) server_negotiation.cc:407] Connection header received
0927 15:11:28.089755 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.089763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.089879 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.090000 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:28.091681 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.092903 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.094534 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.095063 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.095213 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.099796 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:28.099807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:28.099811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:28.099849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:28.102627 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.103487 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.103493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.103495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.103627 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.104076 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.104080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.104082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.104364 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:28.104530 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.105262 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.105498 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260927 15:11:28.106902 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.086751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.2:56735)
0927 15:11:28.087675 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:28.087720 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:28.089462 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:28.090243 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:28.090252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:28.090794 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:28.091498 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.091518 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.093080 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.093084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.094380 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.094393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.095373 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.095380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.095725 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.096492 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:28.096532 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:28.099523 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:28.102788 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:28.102798 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:28.102818 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:28.103348 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:28.103743 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:28.103748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:28.103752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:28.103947 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:28.104493 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:28.104502 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:28.104867 (+ 365us) client_negotiation.cc:769] Sending connection context
0927 15:11:28.105149 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 15:11:28.105727 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":407,"threads_started":1}
I20260927 15:11:28.108541 17878 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34799
I20260927 15:11:28.108865 17878 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:11:28.109622 17878 heartbeater.cc:511] Master 127.0.97.62:34799 requested a full tablet report, sending...
I20260927 15:11:28.111219 17555 ts_manager.cc:194] Registered new tserver with Master: 241a7c8219624719bd141e1065a84a78 (127.0.97.2:45053)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 15:11:28.112175 17555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:11:28.141834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:28.479735 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:28.480060 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:28.480245 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:28.488613 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:28.488744 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:28.488929 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:28.499583 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:28.500938 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:28.502326 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:28.516129 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:28.516088 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:28.516106 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:28.517316 17884 server_base.cc:1034] running on GCE node
I20260927 15:11:28.517935 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:28.518951 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:28.520177 17884 hybrid_clock.cc:648] HybridClock initialized: now 1790521888520127 us; error 36 us; skew 500 ppm
I20260927 15:11:28.520494 17884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:28 dist-test-slave-g0w1 krb5kdc[17510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:28.523787 17884 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:11:28.525282 17884 webserver.cc:460] Webserver started at http://127.0.97.3:40609/ using document root <none> and password file <none>
I20260927 15:11:28.526033 17884 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:28.526134 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:28.526490 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:28.530040 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-g0w1"
server_key: "3039c24950c4a3ff200a37bccc4225c8"
server_key_iv: "543233975f35864ccf4271f63cfcb428"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.530925 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-g0w1"
server_key: "3039c24950c4a3ff200a37bccc4225c8"
server_key_iv: "543233975f35864ccf4271f63cfcb428"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.536620 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:11:28.540701 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.542346 17884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:28.542524 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-g0w1"
server_key: "3039c24950c4a3ff200a37bccc4225c8"
server_key_iv: "543233975f35864ccf4271f63cfcb428"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.542742 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:28.578176 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:28.582614 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:28.582926 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:28.583956 17884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:28.585413 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:28.585493 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.585645 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:28.585697 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.586591 17744 heartbeater.cc:503] Master 127.0.97.62:34799 was elected leader, sending a full tablet report...
I20260927 15:11:28.620458 17884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34071
I20260927 15:11:28.620510 18011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34071 every 8 connection(s)
I20260927 15:11:28.621802 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:28.627501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 17884
I20260927 15:11:28.627691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:28.628159 388 external_mini_cluster.cc:1428] Setting key 1a13e8637aee89d50a201d96e6680fe2
Sep 27 15:11:28 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:28.644831 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.624614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36017 (local address 127.0.97.62:34799)
0927 15:11:28.625128 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.625133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.626312 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 15:11:28.628520 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.628525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.628620 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.628761 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:28.630838 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.631856 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.633589 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.633977 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.634110 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.638026 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:28.638032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:28.638035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:28.638072 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:28.641000 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.642087 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.642093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.642096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.642276 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.642815 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.642820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.642823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.643282 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:28.643431 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.644356 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.644554 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260927 15:11:28.645874 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.625289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34799 (local address 127.0.97.3:36017)
0927 15:11:28.626123 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:28.626178 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:28.628154 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:28.629046 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:28.629060 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:28.629850 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:28.630651 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.630671 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.632027 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.632031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.633448 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.633461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.634209 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.634214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.634443 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.635233 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:28.635263 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:28.637690 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:28.641192 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:28.641202 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:28.641223 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:28.641864 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:28.642392 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:28.642397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:28.642400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:28.642697 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:28.643418 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:28.643428 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:28.643904 (+ 476us) client_negotiation.cc:769] Sending connection context
0927 15:11:28.644266 (+ 362us) client_negotiation.cc:240] Negotiation successful
0927 15:11:28.644699 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":319,"threads_started":1}
I20260927 15:11:28.647324 18012 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34799
I20260927 15:11:28.647667 18012 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:28.648442 18012 heartbeater.cc:511] Master 127.0.97.62:34799 requested a full tablet report, sending...
I20260927 15:11:28.650125 17555 ts_manager.cc:194] Registered new tserver with Master: c2ccff1c970a4fe6bc7f07a40bf5a090 (127.0.97.3:34071)
I20260927 15:11:28.651005 17555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36017
I20260927 15:11:28.656762 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:28.674521 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.661491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.0.97.62:34799)
0927 15:11:28.661666 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.661671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.662178 (+ 507us) server_negotiation.cc:407] Connection header received
0927 15:11:28.662317 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.662321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.662386 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.662474 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:28.664011 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.664778 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.666142 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.666543 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.666643 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.669134 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:28.669138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:28.669141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:28.669176 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:28.671945 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.672668 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.672672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.672674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.672796 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.673131 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.673134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.673136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.673453 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:28.673644 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.674053 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.674261 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:11:28.678937 17555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:28.686129 17555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:28.715655 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.699972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.0.97.2:45053)
0927 15:11:28.705255 (+ 5283us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.705265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.705290 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:11:28.705402 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.705414 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.705703 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.705904 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:28.708204 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.709978 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.713199 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.714425 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.714593 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.714807 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.715105 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.715279 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5109,"thread_start_us":4975,"threads_started":1}
I20260927 15:11:28.715747 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.699831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45053 (local address 127.0.0.1:50828)
0927 15:11:28.700765 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:28.700805 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:28.700987 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:28.706213 (+ 5226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:28.706221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:28.706277 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:28.707985 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.707998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.710131 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.710139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.712980 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.712996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.714626 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.714632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.714821 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.714882 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 15:11:28.715025 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 15:11:28.715279 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":62,"spinlock_wait_cycles":10624,"thread_start_us":742,"threads_started":1}
I20260927 15:11:28.716233 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.705544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34167 (local address 127.0.0.1:48766)
0927 15:11:28.707402 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:28.707418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:28.707571 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:28.708597 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:28.708602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:28.708628 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:28.709002 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.709013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.710828 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.710833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.713201 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.713210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.715258 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.715265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.715464 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.715499 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:11:28.715589 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:11:28.715750 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"mutex_wait_us":110,"thread_start_us":1451,"threads_started":1}
I20260927 15:11:28.716815 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.705665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.97.1:34167)
0927 15:11:28.707873 (+ 2208us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.707883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.707917 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:11:28.708030 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.708039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.708352 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.708601 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:28.709190 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.710707 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.713353 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.714701 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.714852 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.715109 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.716299 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.716498 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":828,"threads_started":1}
I20260927 15:11:28.716850 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.700192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34071 (local address 127.0.0.1:34550)
0927 15:11:28.705109 (+ 4917us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:28.705131 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:28.705356 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:28.706221 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:28.706224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:28.706275 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:28.707653 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.707670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.710214 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.710220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.713627 (+ 3407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:28.713640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.715842 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.715849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:28.716157 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.716200 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:11:28.716287 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:11:28.716440 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4787,"mutex_wait_us":1161,"spinlock_wait_cycles":4864,"thread_start_us":4615,"threads_started":1}
I20260927 15:11:28.717674 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.700399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.0.97.3:34071)
0927 15:11:28.704773 (+ 4374us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.704783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.705233 (+ 450us) server_negotiation.cc:407] Connection header received
0927 15:11:28.705556 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.705563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.705730 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.705899 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:28.707875 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.709404 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.713811 (+ 4407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.715337 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.715466 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.715696 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.717131 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.717333 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4181,"thread_start_us":4026,"threads_started":1}
I20260927 15:11:28.720347 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee34dd4c8ea4657b2be4335f81a2a46 (DEFAULT_TABLE table=test-table [id=ff66ce4e23384c1c8d8b141c2b822292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:28.720513 17813 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee34dd4c8ea4657b2be4335f81a2a46 (DEFAULT_TABLE table=test-table [id=ff66ce4e23384c1c8d8b141c2b822292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:28.721735 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee34dd4c8ea4657b2be4335f81a2a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:28.721956 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee34dd4c8ea4657b2be4335f81a2a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:28.721411 17947 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee34dd4c8ea4657b2be4335f81a2a46 (DEFAULT_TABLE table=test-table [id=ff66ce4e23384c1c8d8b141c2b822292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:28.722797 17947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee34dd4c8ea4657b2be4335f81a2a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:28.734099 18031 tablet_bootstrap.cc:492] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Bootstrap starting.
I20260927 15:11:28.737265 18032 tablet_bootstrap.cc:492] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Bootstrap starting.
I20260927 15:11:28.737792 18031 tablet_bootstrap.cc:654] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:28.739223 18031 log.cc:824] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:28.739253 18033 tablet_bootstrap.cc:492] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Bootstrap starting.
I20260927 15:11:28.741878 18032 tablet_bootstrap.cc:654] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:28.742666 18033 tablet_bootstrap.cc:654] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:28.743505 18032 log.cc:824] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:28.744033 18033 log.cc:824] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:28.746536 18031 tablet_bootstrap.cc:492] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: No bootstrap required, opened a new log
I20260927 15:11:28.746598 18033 tablet_bootstrap.cc:492] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: No bootstrap required, opened a new log
I20260927 15:11:28.746932 18031 ts_tablet_manager.cc:1397] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260927 15:11:28.746970 18033 ts_tablet_manager.cc:1397] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 15:11:28.755309 18033 raft_consensus.cc:348] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:28.755681 18032 tablet_bootstrap.cc:492] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: No bootstrap required, opened a new log
I20260927 15:11:28.755718 18033 raft_consensus.cc:374] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:28.755851 18033 raft_consensus.cc:729] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ccff1c970a4fe6bc7f07a40bf5a090, State: Initialized, Role: FOLLOWER
I20260927 15:11:28.756038 18032 ts_tablet_manager.cc:1397] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20260927 15:11:28.756573 18033 consensus_queue.cc:260] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:28.757763 18012 heartbeater.cc:503] Master 127.0.97.62:34799 was elected leader, sending a full tablet report...
I20260927 15:11:28.758296 18033 ts_tablet_manager.cc:1428] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:11:28.759034 18031 raft_consensus.cc:348] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:28.759654 18031 raft_consensus.cc:374] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:28.759776 18031 raft_consensus.cc:729] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691b65e0c85a454f8fd5733a8ffaaa68, State: Initialized, Role: FOLLOWER
I20260927 15:11:28.760457 18031 consensus_queue.cc:260] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:28.764338 18031 ts_tablet_manager.cc:1428] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Time spent starting tablet: real 0.017s user 0.010s sys 0.009s
I20260927 15:11:28.766861 18032 raft_consensus.cc:348] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:28.767362 18032 raft_consensus.cc:374] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:28.767483 18032 raft_consensus.cc:729] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241a7c8219624719bd141e1065a84a78, State: Initialized, Role: FOLLOWER
I20260927 15:11:28.768142 18032 consensus_queue.cc:260] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:28.769306 17878 heartbeater.cc:503] Master 127.0.97.62:34799 was elected leader, sending a full tablet report...
I20260927 15:11:28.769935 18032 ts_tablet_manager.cc:1428] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
W20260927 15:11:28.806423 17745 tablet.cc:2369] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:28.835345 17879 tablet.cc:2369] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:28.873628 18013 tablet.cc:2369] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:29.028259 18039 raft_consensus.cc:482] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:29.028626 18039 raft_consensus.cc:504] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:29.030431 18039 leader_election.cc:288] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ccff1c970a4fe6bc7f07a40bf5a090 (127.0.97.3:34071), 691b65e0c85a454f8fd5733a8ffaaa68 (127.0.97.1:34167)
I20260927 15:11:29.038347 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.032127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35381 (local address 127.0.97.1:34167)
0927 15:11:29.032341 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.032347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.032630 (+ 283us) server_negotiation.cc:407] Connection header received
0927 15:11:29.032795 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.032800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.032892 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.033002 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:29.033763 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.034561 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.036470 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.037298 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.037495 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.037609 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.037923 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.038071 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:11:29.038378 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.031136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60127 (local address 127.0.97.3:34071)
0927 15:11:29.031390 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.031399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.032175 (+ 776us) server_negotiation.cc:407] Connection header received
0927 15:11:29.032547 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.032556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.032645 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.032748 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:29.033496 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.034327 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.036411 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.037297 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.037436 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.037591 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.037982 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.038113 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:11:29.038550 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.030935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34071 (local address 127.0.97.2:60127)
0927 15:11:29.032040 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:29.032064 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:29.032321 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:29.032790 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:29.032795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:29.032823 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:29.033312 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:29.033324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.034493 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.034499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:29.036220 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:29.036235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.037551 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.037561 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:29.037784 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.037826 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:11:29.037930 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:11:29.038073 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":9,"thread_start_us":450,"threads_started":1}
I20260927 15:11:29.038578 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.031840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34167 (local address 127.0.97.2:35381)
0927 15:11:29.032513 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:29.032531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:29.032660 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:29.033047 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:29.033054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:29.033077 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:29.033589 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:29.033603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.034731 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.034735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:29.036255 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:29.036268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.037578 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.037584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:29.037769 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.037795 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:11:29.037872 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:11:29.038042 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":23,"thread_start_us":323,"threads_started":1}
I20260927 15:11:29.039973 17967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee34dd4c8ea4657b2be4335f81a2a46" candidate_uuid: "241a7c8219624719bd141e1065a84a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" is_pre_election: true
I20260927 15:11:29.040035 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee34dd4c8ea4657b2be4335f81a2a46" candidate_uuid: "241a7c8219624719bd141e1065a84a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" is_pre_election: true
I20260927 15:11:29.040542 17699 raft_consensus.cc:2454] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 241a7c8219624719bd141e1065a84a78 in term 0.
I20260927 15:11:29.040548 17967 raft_consensus.cc:2454] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 241a7c8219624719bd141e1065a84a78 in term 0.
I20260927 15:11:29.041237 17769 leader_election.cc:302] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 241a7c8219624719bd141e1065a84a78, c2ccff1c970a4fe6bc7f07a40bf5a090; no voters:
I20260927 15:11:29.041642 18039 raft_consensus.cc:2781] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:29.041792 18039 raft_consensus.cc:482] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:29.041930 18039 raft_consensus.cc:3037] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:29.044327 18039 raft_consensus.cc:504] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:29.045014 18039 leader_election.cc:288] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [CANDIDATE]: Term 1 election: Requested vote from peers c2ccff1c970a4fe6bc7f07a40bf5a090 (127.0.97.3:34071), 691b65e0c85a454f8fd5733a8ffaaa68 (127.0.97.1:34167)
I20260927 15:11:29.045657 17967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee34dd4c8ea4657b2be4335f81a2a46" candidate_uuid: "241a7c8219624719bd141e1065a84a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090"
I20260927 15:11:29.045759 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee34dd4c8ea4657b2be4335f81a2a46" candidate_uuid: "241a7c8219624719bd141e1065a84a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691b65e0c85a454f8fd5733a8ffaaa68"
I20260927 15:11:29.045877 17967 raft_consensus.cc:3037] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:29.046066 17699 raft_consensus.cc:3037] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:29.048534 17699 raft_consensus.cc:2454] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 241a7c8219624719bd141e1065a84a78 in term 1.
I20260927 15:11:29.048532 17967 raft_consensus.cc:2454] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 241a7c8219624719bd141e1065a84a78 in term 1.
I20260927 15:11:29.049052 17769 leader_election.cc:302] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 241a7c8219624719bd141e1065a84a78, c2ccff1c970a4fe6bc7f07a40bf5a090; no voters:
I20260927 15:11:29.049489 18039 raft_consensus.cc:2781] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:29.049994 18039 raft_consensus.cc:686] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 1 LEADER]: Becoming Leader. State: Replica: 241a7c8219624719bd141e1065a84a78, State: Running, Role: LEADER
I20260927 15:11:29.050498 18039 consensus_queue.cc:237] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } }
I20260927 15:11:29.055714 17555 catalog_manager.cc:5495] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 241a7c8219624719bd141e1065a84a78 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "241a7c8219624719bd141e1065a84a78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241a7c8219624719bd141e1065a84a78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:29.081833 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.076150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.97.2:45053)
0927 15:11:29.076365 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.076370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.076385 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:11:29.076485 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.076489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.076576 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.076740 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:29.077171 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.077918 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.079459 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.079909 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.080057 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.080604 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:29.081164 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:29.081349 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.081487 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.081568 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:11:29.092761 17699 raft_consensus.cc:1261] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 1 FOLLOWER]: Refusing update from remote peer 241a7c8219624719bd141e1065a84a78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:29.092804 17967 raft_consensus.cc:1261] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 1 FOLLOWER]: Refusing update from remote peer 241a7c8219624719bd141e1065a84a78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:29.093870 18039 consensus_queue.cc:1035] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691b65e0c85a454f8fd5733a8ffaaa68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:29.094278 18042 consensus_queue.cc:1035] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ccff1c970a4fe6bc7f07a40bf5a090" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:29.107626 18045 mvcc.cc:205] Tried to move back new op lower bound from 7333977657711390720 to 7333977657555255296. Current Snapshot: MvccSnapshot[applied={T|T < 7333977657711390720}]
I20260927 15:11:29.110510 18047 mvcc.cc:205] Tried to move back new op lower bound from 7333977657711390720 to 7333977657555255296. Current Snapshot: MvccSnapshot[applied={T|T < 7333977657711390720}]
I20260927 15:11:29.113343 18046 mvcc.cc:205] Tried to move back new op lower bound from 7333977657711390720 to 7333977657555255296. Current Snapshot: MvccSnapshot[applied={T|T < 7333977657711390720}]
I20260927 15:11:29.122205 17555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:29.122503 17555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:29.126588 17555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d65f9f87e0d449f893f9343bcbf2e46: Sending DeleteTablet for 3 replicas of tablet 5ee34dd4c8ea4657b2be4335f81a2a46
I20260927 15:11:29.128121 17813 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee34dd4c8ea4657b2be4335f81a2a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50828
I20260927 15:11:29.128381 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee34dd4c8ea4657b2be4335f81a2a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48766
I20260927 15:11:29.128381 17947 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee34dd4c8ea4657b2be4335f81a2a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34550
I20260927 15:11:29.131060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17616
I20260927 15:11:29.138595 18055 tablet_replica.cc:330] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: stopping tablet replica
I20260927 15:11:29.139257 18055 raft_consensus.cc:2229] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:29.140187 18055 raft_consensus.cc:2258] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:29.143457 18056 tablet_replica.cc:330] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: stopping tablet replica
I20260927 15:11:29.144048 18056 raft_consensus.cc:2229] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:29.144693 18056 raft_consensus.cc:2258] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:29.147248 18055 ts_tablet_manager.cc:1905] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:29.147758 18056 ts_tablet_manager.cc:1905] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:29.155898 18055 ts_tablet_manager.cc:1918] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:29.156260 18055 log.cc:1196] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/5ee34dd4c8ea4657b2be4335f81a2a46
I20260927 15:11:29.157006 18055 ts_tablet_manager.cc:1939] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 241a7c8219624719bd141e1065a84a78: Deleting consensus metadata
I20260927 15:11:29.157135 18056 ts_tablet_manager.cc:1918] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:29.157347 18056 log.cc:1196] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/5ee34dd4c8ea4657b2be4335f81a2a46
I20260927 15:11:29.158018 18056 ts_tablet_manager.cc:1939] T 5ee34dd4c8ea4657b2be4335f81a2a46 P 691b65e0c85a454f8fd5733a8ffaaa68: Deleting consensus metadata
I20260927 15:11:29.159516 17540 catalog_manager.cc:4841] TS 241a7c8219624719bd141e1065a84a78 (127.0.97.2:45053): tablet 5ee34dd4c8ea4657b2be4335f81a2a46 (table test-table [id=ff66ce4e23384c1c8d8b141c2b822292]) successfully deleted
I20260927 15:11:29.161015 17542 catalog_manager.cc:4841] TS 691b65e0c85a454f8fd5733a8ffaaa68 (127.0.97.1:34167): tablet 5ee34dd4c8ea4657b2be4335f81a2a46 (table test-table [id=ff66ce4e23384c1c8d8b141c2b822292]) successfully deleted
I20260927 15:11:29.161789 18057 tablet_replica.cc:330] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: stopping tablet replica
I20260927 15:11:29.162249 18057 raft_consensus.cc:2229] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:29.162709 18057 raft_consensus.cc:2258] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:29.164852 18057 ts_tablet_manager.cc:1905] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:11:29 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:11:29.170845 18057 ts_tablet_manager.cc:1918] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:29.171043 18057 log.cc:1196] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/5ee34dd4c8ea4657b2be4335f81a2a46
I20260927 15:11:29.171664 18057 ts_tablet_manager.cc:1939] T 5ee34dd4c8ea4657b2be4335f81a2a46 P c2ccff1c970a4fe6bc7f07a40bf5a090: Deleting consensus metadata
I20260927 15:11:29.173496 17542 catalog_manager.cc:4841] TS c2ccff1c970a4fe6bc7f07a40bf5a090 (127.0.97.3:34071): tablet 5ee34dd4c8ea4657b2be4335f81a2a46 (table test-table [id=ff66ce4e23384c1c8d8b141c2b822292]) successfully deleted
I20260927 15:11:29.178059 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.133123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.97.1:34167)
0927 15:11:29.133374 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.133382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.157728 (+ 24346us) server_negotiation.cc:407] Connection header received
0927 15:11:29.158328 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.158334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.158417 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.158495 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:29.160661 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.161842 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.163926 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.164569 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.164715 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.170371 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:29.170391 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:29.170419 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:29.170487 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:29.174699 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:29.175783 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:29.175793 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:29.175795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:29.175970 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:29.176468 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:29.176472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:29.176474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:29.176844 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:29.177093 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.177537 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.177725 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:11:29.178737 17739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:29.320982 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17616
I20260927 15:11:29.339946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17750
Sep 27 15:11:29 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:11:29.357213 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.340942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.97.2:45053)
0927 15:11:29.341139 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.341146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.341199 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:11:29.341275 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.341284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.341365 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.341481 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:29.342868 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.343925 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.345346 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.345870 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.346005 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.350611 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:29.350628 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:29.350646 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:29.350687 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:29.354018 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:29.354849 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:29.354857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:29.354859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:29.355023 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:29.355545 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:29.355553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:29.355559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:29.355950 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:29.356305 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.356592 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.356844 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:11:29.357977 17873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:29.478524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17750
I20260927 15:11:29.497975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17884
Sep 27 15:11:29 dist-test-slave-g0w1 krb5kdc[17510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:11:29.514412 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.498734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.0.97.3:34071)
0927 15:11:29.498903 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.498910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.499022 (+ 112us) server_negotiation.cc:407] Connection header received
0927 15:11:29.499239 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.499248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.499340 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.499436 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:29.500809 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.501544 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.502819 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.503288 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.503389 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.507861 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:29.507923 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:29.507940 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:29.507989 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:29.510987 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:29.511922 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:29.511955 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:29.511961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:29.512195 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:29.512682 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:29.512686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:29.512689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:29.513174 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:29.513418 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.513786 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.514019 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:29.515230 18007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:29.606523 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17884
I20260927 15:11:29.623929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 17523
I20260927 15:11:29.625213 17585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:29.739140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 17523
2026-09-27T15:11:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4710 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:29Z Disabled control of system clock
I20260927 15:11:29.822206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33497
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:30.176294 18066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:30.176630 18066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:30.176717 18066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:30.184383 18066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:30.184515 18066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:30.184562 18066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:30.184614 18066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:30.184660 18066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:30.194185 18066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:30.195539 18066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:30.197465 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:30.210121 18071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:30.210108 18072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:30.210371 18074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:30.210930 18066 server_base.cc:1034] running on GCE node
I20260927 15:11:30.211815 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:30.213373 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:30.214614 18066 hybrid_clock.cc:648] HybridClock initialized: now 1790521890214547 us; error 70 us; skew 500 ppm
I20260927 15:11:30.214986 18066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:30.223694 18066 webserver.cc:460] Webserver started at http://127.0.97.62:39125/ using document root <none> and password file <none>
I20260927 15:11:30.224491 18066 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:30.224696 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:30.225013 18066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:30.228154 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "1c0a12f3edfc47e1815095eea18d40c9"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-g0w1"
server_key: "c19d22ee3f0549f41d5b903e8a7bdc1a"
server_key_iv: "ebb0de9dd07d6bd23ae0fa1af1d8f97f"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.228974 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c0a12f3edfc47e1815095eea18d40c9"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-g0w1"
server_key: "c19d22ee3f0549f41d5b903e8a7bdc1a"
server_key_iv: "ebb0de9dd07d6bd23ae0fa1af1d8f97f"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.234726 18066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:11:30.238873 18080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:30.240589 18066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:30.240811 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "1c0a12f3edfc47e1815095eea18d40c9"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-g0w1"
server_key: "c19d22ee3f0549f41d5b903e8a7bdc1a"
server_key_iv: "ebb0de9dd07d6bd23ae0fa1af1d8f97f"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.241070 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:30.273916 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:30.274896 18066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:30.275243 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:30.294878 18066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33497
I20260927 15:11:30.294899 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33497 every 8 connection(s)
I20260927 15:11:30.296317 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:30.297868 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18066
I20260927 15:11:30.298054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:30.298636 388 external_mini_cluster.cc:1428] Setting key ebb708c4152f63de3771ba14a051f630
I20260927 15:11:30.301468 18132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:30.315026 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:30.300741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.0.97.62:33497)
0927 15:11:30.301870 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:30.301882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:30.301958 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:11:30.303679 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:30.303702 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:30.304217 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:30.304632 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:30.305217 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:30.306578 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:30.308110 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:30.309309 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:30.309516 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:30.310993 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:30.311011 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:30.311034 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:30.312008 (+ 974us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:30.312122 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:30.312157 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:30.312294 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:30.313044 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 15:11:30.313511 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":394,"threads_started":1}
I20260927 15:11:30.315346 18132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9: Bootstrap starting.
I20260927 15:11:30.319245 18132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:30.320516 18132 log.cc:824] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:30.324510 18132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9: No bootstrap required, opened a new log
I20260927 15:11:30.333877 18132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0a12f3edfc47e1815095eea18d40c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260927 15:11:30.334335 18132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:30.334450 18132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0a12f3edfc47e1815095eea18d40c9, State: Initialized, Role: FOLLOWER
I20260927 15:11:30.335143 18132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0a12f3edfc47e1815095eea18d40c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260927 15:11:30.335412 18132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:30.335561 18132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:30.335749 18132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:30.338052 18132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0a12f3edfc47e1815095eea18d40c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260927 15:11:30.338644 18132 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0a12f3edfc47e1815095eea18d40c9; no voters:
I20260927 15:11:30.339536 18132 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:30.339716 18137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:30.340512 18137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0a12f3edfc47e1815095eea18d40c9, State: Running, Role: LEADER
I20260927 15:11:30.341099 18137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0a12f3edfc47e1815095eea18d40c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260927 15:11:30.341773 18132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:30.343948 18139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0a12f3edfc47e1815095eea18d40c9. Latest consensus state: current_term: 1 leader_uuid: "1c0a12f3edfc47e1815095eea18d40c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0a12f3edfc47e1815095eea18d40c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } } }
I20260927 15:11:30.344192 18138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c0a12f3edfc47e1815095eea18d40c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0a12f3edfc47e1815095eea18d40c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } } }
I20260927 15:11:30.344444 18139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:30.345369 18138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:30.348014 18144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:30.352531 18144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:30.361353 18144 catalog_manager.cc:1347] Generated new cluster ID: e2d3af24943c4ebdb4b146d1473d6326
I20260927 15:11:30.361477 18144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:30.385063 18144 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:30.386551 18144 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:30.397903 18144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9: Generated new TSK 0
I20260927 15:11:30.399160 18144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:30.418426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33497
--builtin_ntp_servers=127.0.97.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:30.773267 18156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:30.773658 18156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:30.773828 18156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:30.781796 18156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:30.782145 18156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:30.782351 18156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:30.792618 18156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:30.794049 18156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:30.796123 18156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:30.811113 18162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:30.811252 18161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:30.811036 18164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:30.811873 18156 server_base.cc:1034] running on GCE node
I20260927 15:11:30.812719 18156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:30.813738 18156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:30.815042 18156 hybrid_clock.cc:648] HybridClock initialized: now 1790521890814983 us; error 43 us; skew 500 ppm
I20260927 15:11:30.815539 18156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:30.818526 18156 webserver.cc:460] Webserver started at http://127.0.97.1:40823/ using document root <none> and password file <none>
I20260927 15:11:30.819375 18156 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:30.819494 18156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:30.819832 18156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:30.822893 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "c7eb06ab221a4842bb864d5f2e18e9bf"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-g0w1"
server_key: "ce6f413339fda7d7d62ba40561c655af"
server_key_iv: "42855f7a7c83d2e0e08d5dc73c945160"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.823845 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c7eb06ab221a4842bb864d5f2e18e9bf"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-g0w1"
server_key: "ce6f413339fda7d7d62ba40561c655af"
server_key_iv: "42855f7a7c83d2e0e08d5dc73c945160"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.829703 18156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:11:30.833807 18170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:30.835956 18156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:30.836161 18156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "c7eb06ab221a4842bb864d5f2e18e9bf"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-g0w1"
server_key: "ce6f413339fda7d7d62ba40561c655af"
server_key_iv: "42855f7a7c83d2e0e08d5dc73c945160"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.836421 18156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:30.861872 18156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:30.862943 18156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:30.863349 18156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:30.864480 18156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:30.866268 18156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:30.866405 18156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:30.866528 18156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:30.866626 18156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:30.904822 18156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43607
I20260927 15:11:30.904903 18282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43607 every 8 connection(s)
I20260927 15:11:30.906286 18156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:30.907410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18156
I20260927 15:11:30.907590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:30.908054 388 external_mini_cluster.cc:1428] Setting key e4456b1913d78dfdfc018e2f4bec7f85
I20260927 15:11:30.911587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33497
--builtin_ntp_servers=127.0.97.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:30.924582 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:30.909281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33391 (local address 127.0.97.62:33497)
0927 15:11:30.909808 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:30.909813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:30.911621 (+ 1808us) server_negotiation.cc:407] Connection header received
0927 15:11:30.915147 (+ 3526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:30.915153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:30.915244 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:30.915346 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:30.917080 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:30.917858 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:30.920858 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:30.921325 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:30.921444 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:30.923464 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:30.923473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:30.923481 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:30.923513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:30.923581 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:30.923601 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:30.923711 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:30.923946 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 15:11:30.924077 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
I20260927 15:11:30.925585 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:30.910075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33497 (local address 127.0.97.1:33391)
0927 15:11:30.911386 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:30.911454 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:30.914633 (+ 3179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:30.915694 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:30.915711 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:30.915871 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:30.916845 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:30.916878 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:30.918043 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:30.918053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:30.920646 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:30.920673 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:30.921629 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:30.921639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:30.922052 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:30.923020 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:30.923023 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:30.923109 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:30.923217 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:30.923717 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:30.923725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:30.923732 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:11:30.923896 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 15:11:30.924296 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":360,"threads_started":1}
I20260927 15:11:30.927596 18283 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33497
I20260927 15:11:30.927970 18283 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:30.928790 18283 heartbeater.cc:511] Master 127.0.97.62:33497 requested a full tablet report, sending...
I20260927 15:11:30.931448 18097 ts_manager.cc:194] Registered new tserver with Master: c7eb06ab221a4842bb864d5f2e18e9bf (127.0.97.1:43607)
I20260927 15:11:30.933091 18097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33391
W20260927 15:11:31.276225 18287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:31.276539 18287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:31.276682 18287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:31.284456 18287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:31.284577 18287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:31.284756 18287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:31.295027 18287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:31.296751 18287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:31.299252 18287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:31.315532 18292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:31.315587 18293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:31.315819 18295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:31.316340 18287 server_base.cc:1034] running on GCE node
I20260927 15:11:31.317003 18287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:31.318053 18287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:31.319285 18287 hybrid_clock.cc:648] HybridClock initialized: now 1790521891319226 us; error 51 us; skew 500 ppm
I20260927 15:11:31.319664 18287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:31.322207 18287 webserver.cc:460] Webserver started at http://127.0.97.2:42351/ using document root <none> and password file <none>
I20260927 15:11:31.323103 18287 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:31.323240 18287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:31.323654 18287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:31.326941 18287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-g0w1"
server_key: "891f2dba1db70f24d4019da770a10d7a"
server_key_iv: "2335b5844c224a59b9db3bb67d631740"
server_key_version: "encryptionkey@0"
I20260927 15:11:31.327947 18287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-g0w1"
server_key: "891f2dba1db70f24d4019da770a10d7a"
server_key_iv: "2335b5844c224a59b9db3bb67d631740"
server_key_version: "encryptionkey@0"
I20260927 15:11:31.333855 18287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:11:31.337888 18301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:31.339941 18287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:11:31.340161 18287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-g0w1"
server_key: "891f2dba1db70f24d4019da770a10d7a"
server_key_iv: "2335b5844c224a59b9db3bb67d631740"
server_key_version: "encryptionkey@0"
I20260927 15:11:31.340423 18287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:31.394488 18287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:31.395483 18287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:31.395783 18287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:31.396823 18287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:31.398406 18287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:31.398528 18287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:31.398664 18287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:31.398753 18287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:31.436460 18287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40421
I20260927 15:11:31.436518 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40421 every 8 connection(s)
I20260927 15:11:31.438014 18287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:31.444571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18287
I20260927 15:11:31.444769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:31.445398 388 external_mini_cluster.cc:1428] Setting key a3350790379d250efe2bb78d5a8b2750
I20260927 15:11:31.448760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33497
--builtin_ntp_servers=127.0.97.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:31.452672 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:31.440895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47057 (local address 127.0.97.62:33497)
0927 15:11:31.441410 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:31.441415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:31.442733 (+ 1318us) server_negotiation.cc:407] Connection header received
0927 15:11:31.445066 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:31.445071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:31.445158 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:31.445262 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:31.446906 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.447907 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.449691 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.450106 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.450245 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:31.451761 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:31.451766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:31.451770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:31.451791 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:31.451843 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:31.451855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:31.451949 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:31.452179 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:11:31.452288 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
I20260927 15:11:31.453754 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:31.441552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33497 (local address 127.0.97.2:47057)
0927 15:11:31.442570 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:31.442612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:31.444706 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:31.445615 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:31.445628 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:31.445775 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:31.446729 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:31.446750 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.448064 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.448069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:31.449534 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:31.449548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.450345 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.450353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:31.450608 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:31.451404 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:31.451407 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:31.451451 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:31.451545 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:31.451956 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:31.451969 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:31.451981 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:11:31.452169 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:11:31.452629 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":341,"threads_started":1}
I20260927 15:11:31.455361 18414 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33497
I20260927 15:11:31.455768 18414 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:31.456784 18414 heartbeater.cc:511] Master 127.0.97.62:33497 requested a full tablet report, sending...
I20260927 15:11:31.458448 18097 ts_manager.cc:194] Registered new tserver with Master: 3bc397e59b1b408f9ffcaa90f19c2b5f (127.0.97.2:40421)
I20260927 15:11:31.459753 18097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:47057
W20260927 15:11:31.809274 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:31.809569 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:31.809653 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:31.817569 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:31.817692 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:31.817800 18418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:31.828217 18418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:31.829461 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:31.831303 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:31.845479 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:31.845705 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:31.845705 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:31.847340 18418 server_base.cc:1034] running on GCE node
I20260927 15:11:31.847961 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:31.848987 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:31.850244 18418 hybrid_clock.cc:648] HybridClock initialized: now 1790521891850178 us; error 45 us; skew 500 ppm
I20260927 15:11:31.850675 18418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:31.853334 18418 webserver.cc:460] Webserver started at http://127.0.97.3:33631/ using document root <none> and password file <none>
I20260927 15:11:31.854173 18418 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:31.854307 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:31.854692 18418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:31.858125 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "00e1b972f516418c98271614df46576a"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-g0w1"
server_key: "c6c79fcdcbdbea74727c3418f6b290ed"
server_key_iv: "7ab27cdf88ba020ca86b1bd3cba3cc50"
server_key_version: "encryptionkey@0"
I20260927 15:11:31.859063 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "00e1b972f516418c98271614df46576a"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-g0w1"
server_key: "c6c79fcdcbdbea74727c3418f6b290ed"
server_key_iv: "7ab27cdf88ba020ca86b1bd3cba3cc50"
server_key_version: "encryptionkey@0"
I20260927 15:11:31.865034 18418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:11:31.869251 18432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:31.870863 18418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:31.871033 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "00e1b972f516418c98271614df46576a"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-g0w1"
server_key: "c6c79fcdcbdbea74727c3418f6b290ed"
server_key_iv: "7ab27cdf88ba020ca86b1bd3cba3cc50"
server_key_version: "encryptionkey@0"
I20260927 15:11:31.871345 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:31.909420 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:31.910317 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:31.910578 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:31.911696 18418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:31.913357 18418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:31.913468 18418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:31.913563 18418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:31.913617 18418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:31.936722 18283 heartbeater.cc:503] Master 127.0.97.62:33497 was elected leader, sending a full tablet report...
I20260927 15:11:31.952708 18418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39583
I20260927 15:11:31.952816 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39583 every 8 connection(s)
I20260927 15:11:31.954172 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:31.957305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18418
I20260927 15:11:31.957516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:31.958209 388 external_mini_cluster.cc:1428] Setting key ecedb5e7e1f1c05e58561e32dc98bac7
I20260927 15:11:31.972769 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:31.959538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46821 (local address 127.0.97.62:33497)
0927 15:11:31.964514 (+ 4976us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:31.964518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:31.964533 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:11:31.964582 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:31.964585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:31.964656 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:31.964743 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:31.966758 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.967797 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.969719 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.970176 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.970286 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:31.971920 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:31.971926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:31.971930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:31.971985 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:31.972026 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:31.972040 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:31.972119 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:31.972347 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 15:11:31.972441 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4827,"thread_start_us":4682,"threads_started":1}
I20260927 15:11:31.973922 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:31.958816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33497 (local address 127.0.97.3:46821)
0927 15:11:31.961059 (+ 2243us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:31.961186 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:31.963800 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:31.965137 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:31.965152 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:31.965316 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:31.966435 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:31.966488 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.968017 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.968022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:31.969537 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:31.969554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.970400 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.970406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:31.970643 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:31.971573 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:31.971575 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:31.971623 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:31.971714 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:31.972138 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:31.972147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:31.972154 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:11:31.972321 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 15:11:31.972679 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"thread_start_us":1101,"threads_started":1}
I20260927 15:11:31.975835 18545 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33497
I20260927 15:11:31.976246 18545 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:31.977212 18545 heartbeater.cc:511] Master 127.0.97.62:33497 requested a full tablet report, sending...
I20260927 15:11:31.978816 18096 ts_manager.cc:194] Registered new tserver with Master: 00e1b972f516418c98271614df46576a (127.0.97.3:39583)
I20260927 15:11:31.979902 18096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46821
I20260927 15:11:31.990156 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:32.002501 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:31.994876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.0.97.62:33497)
0927 15:11:31.995057 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:31.995062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:31.995592 (+ 530us) server_negotiation.cc:407] Connection header received
0927 15:11:31.995766 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:31.995771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:31.995889 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:31.996002 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:31.996595 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:31.997387 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:31.999333 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.000148 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.000295 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.001609 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:32.001619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:32.001625 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:32.001654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:32.001700 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:32.001718 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:32.001824 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.002021 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.002173 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:11:32.007279 18096 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:32.014482 18096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:32.049932 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.028719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43607 (local address 127.0.0.1:55608)
0927 15:11:32.040466 (+ 11747us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:32.040481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:32.040699 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:32.041729 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:32.041737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:32.041765 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:32.042168 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.042186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.044008 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.044015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.046765 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.046782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.048880 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.048887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.049111 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.049163 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:11:32.049310 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:11:32.049525 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11630,"mutex_wait_us":64,"thread_start_us":11157,"threads_started":1}
I20260927 15:11:32.049957 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.028749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.0.97.1:43607)
0927 15:11:32.039725 (+ 10976us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.039739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.040615 (+ 876us) server_negotiation.cc:407] Connection header received
0927 15:11:32.040888 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.040900 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.041201 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.041421 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:32.042374 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.043876 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.047040 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.048649 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.048799 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.049109 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.049384 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.049597 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10695,"thread_start_us":10990,"threads_started":1}
I20260927 15:11:32.051115 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.027881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.0.97.2:40421)
0927 15:11:32.041423 (+ 13542us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.041435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.041467 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:11:32.041566 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.041575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.041839 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.042053 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:32.042975 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.044748 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.047655 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.048778 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.048914 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.049158 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.050195 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.050722 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13303,"thread_start_us":13108,"threads_started":1}
I20260927 15:11:32.050046 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.027633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40421 (local address 127.0.0.1:47574)
0927 15:11:32.038993 (+ 11360us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:32.039054 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:32.041063 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:32.042194 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:32.042199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:32.042226 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:32.042744 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.042760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.044869 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.044877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.047436 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.047451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.049296 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.049304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.049485 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.049518 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:11:32.049600 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:11:32.049727 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11045,"mutex_wait_us":28,"thread_start_us":12530,"threads_started":1}
I20260927 15:11:32.051760 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.028681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.97.3:39583)
0927 15:11:32.041769 (+ 13088us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.041778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.041814 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:11:32.041958 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.041964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.042669 (+ 705us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.042884 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:32.043971 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.045525 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.048046 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.049487 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.049627 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.049896 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.051207 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.051381 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12905,"thread_start_us":12794,"threads_started":1}
I20260927 15:11:32.052400 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.028067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39583 (local address 127.0.0.1:44914)
0927 15:11:32.039511 (+ 11444us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:32.039531 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:32.040306 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:32.043170 (+ 2864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:32.043173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:32.043198 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:32.043603 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.043616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.045669 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.045677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.047817 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.047833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.050059 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.050545 (+ 486us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.050736 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.050778 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:11:32.050889 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:11:32.051030 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11303,"mutex_wait_us":4,"thread_start_us":10489,"threads_started":1}
I20260927 15:11:32.054804 18218 tablet_service.cc:1467] Processing CreateTablet for tablet 46df5e995a2645b4905a2977170cd735 (DEFAULT_TABLE table=test-table [id=fc7bce703bf04f1bbc421bee5c08283e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:32.055346 18349 tablet_service.cc:1467] Processing CreateTablet for tablet 46df5e995a2645b4905a2977170cd735 (DEFAULT_TABLE table=test-table [id=fc7bce703bf04f1bbc421bee5c08283e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:32.055505 18480 tablet_service.cc:1467] Processing CreateTablet for tablet 46df5e995a2645b4905a2977170cd735 (DEFAULT_TABLE table=test-table [id=fc7bce703bf04f1bbc421bee5c08283e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:32.056551 18218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df5e995a2645b4905a2977170cd735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:32.056694 18349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df5e995a2645b4905a2977170cd735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:32.056896 18480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df5e995a2645b4905a2977170cd735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:32.068789 18564 tablet_bootstrap.cc:492] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Bootstrap starting.
I20260927 15:11:32.072026 18565 tablet_bootstrap.cc:492] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Bootstrap starting.
I20260927 15:11:32.073370 18564 tablet_bootstrap.cc:654] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:32.073846 18566 tablet_bootstrap.cc:492] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Bootstrap starting.
I20260927 15:11:32.074889 18564 log.cc:824] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:32.076030 18565 tablet_bootstrap.cc:654] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:32.077322 18565 log.cc:824] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:32.077394 18566 tablet_bootstrap.cc:654] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:32.078630 18564 tablet_bootstrap.cc:492] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: No bootstrap required, opened a new log
I20260927 15:11:32.078614 18566 log.cc:824] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:32.079131 18564 ts_tablet_manager.cc:1397] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260927 15:11:32.081269 18566 tablet_bootstrap.cc:492] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: No bootstrap required, opened a new log
I20260927 15:11:32.081553 18566 ts_tablet_manager.cc:1397] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:11:32.088498 18565 tablet_bootstrap.cc:492] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: No bootstrap required, opened a new log
I20260927 15:11:32.088828 18565 ts_tablet_manager.cc:1397] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260927 15:11:32.090075 18566 raft_consensus.cc:348] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.090502 18566 raft_consensus.cc:374] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:32.090643 18566 raft_consensus.cc:729] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7eb06ab221a4842bb864d5f2e18e9bf, State: Initialized, Role: FOLLOWER
I20260927 15:11:32.091346 18566 consensus_queue.cc:260] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.091328 18564 raft_consensus.cc:348] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.091784 18564 raft_consensus.cc:374] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:32.091959 18564 raft_consensus.cc:729] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e1b972f516418c98271614df46576a, State: Initialized, Role: FOLLOWER
I20260927 15:11:32.092818 18564 consensus_queue.cc:260] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.094000 18545 heartbeater.cc:503] Master 127.0.97.62:33497 was elected leader, sending a full tablet report...
I20260927 15:11:32.096194 18566 ts_tablet_manager.cc:1428] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 15:11:32.097738 18564 ts_tablet_manager.cc:1428] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260927 15:11:32.099512 18565 raft_consensus.cc:348] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.100023 18565 raft_consensus.cc:374] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:32.100134 18565 raft_consensus.cc:729] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc397e59b1b408f9ffcaa90f19c2b5f, State: Initialized, Role: FOLLOWER
I20260927 15:11:32.100718 18565 consensus_queue.cc:260] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.101804 18414 heartbeater.cc:503] Master 127.0.97.62:33497 was elected leader, sending a full tablet report...
I20260927 15:11:32.102397 18565 ts_tablet_manager.cc:1428] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260927 15:11:32.159300 18284 tablet.cc:2369] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:32.184368 18572 raft_consensus.cc:482] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:32.184768 18572 raft_consensus.cc:504] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.186560 18572 leader_election.cc:288] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e1b972f516418c98271614df46576a (127.0.97.3:39583), c7eb06ab221a4842bb864d5f2e18e9bf (127.0.97.1:43607)
W20260927 15:11:32.190361 18415 tablet.cc:2369] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:32.194806 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.187222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35193 (local address 127.0.97.1:43607)
0927 15:11:32.187410 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.187416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.188220 (+ 804us) server_negotiation.cc:407] Connection header received
0927 15:11:32.188434 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.188440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.188540 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.188650 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:32.189532 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.190612 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.193045 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.193840 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.193987 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.194109 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.194418 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.194546 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:11:32.194832 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.187189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54073 (local address 127.0.97.3:39583)
0927 15:11:32.187398 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.187405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.188220 (+ 815us) server_negotiation.cc:407] Connection header received
0927 15:11:32.188455 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.188461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.188539 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.188649 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:32.189476 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.190616 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.193106 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.193891 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.194053 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.194158 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.194459 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.194591 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:11:32.194969 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.187007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43607 (local address 127.0.97.2:35193)
0927 15:11:32.188082 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:32.188097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:32.188261 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:32.188689 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:32.188694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:32.188721 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:32.189365 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.189378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.190762 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.190768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.192812 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.192826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.194061 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.194068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.194246 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.194272 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:11:32.194378 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:11:32.194517 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":142,"thread_start_us":620,"threads_started":1}
I20260927 15:11:32.195143 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.186974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39583 (local address 127.0.97.2:54073)
0927 15:11:32.188057 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:32.188082 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:32.188294 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:32.188689 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:32.188694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:32.188734 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:32.189270 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.189292 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.190843 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.190851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.192801 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:32.192817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.194101 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.194108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:32.194270 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.194293 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:11:32.194398 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:11:32.194553 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":130,"spinlock_wait_cycles":51200,"thread_start_us":686,"threads_started":1}
I20260927 15:11:32.196062 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df5e995a2645b4905a2977170cd735" candidate_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" is_pre_election: true
I20260927 15:11:32.196059 18500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df5e995a2645b4905a2977170cd735" candidate_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e1b972f516418c98271614df46576a" is_pre_election: true
I20260927 15:11:32.196566 18500 raft_consensus.cc:2454] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc397e59b1b408f9ffcaa90f19c2b5f in term 0.
I20260927 15:11:32.196566 18238 raft_consensus.cc:2454] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc397e59b1b408f9ffcaa90f19c2b5f in term 0.
I20260927 15:11:32.197404 18305 leader_election.cc:302] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00e1b972f516418c98271614df46576a, 3bc397e59b1b408f9ffcaa90f19c2b5f, c7eb06ab221a4842bb864d5f2e18e9bf; no voters:
I20260927 15:11:32.197877 18572 raft_consensus.cc:2781] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:32.198035 18572 raft_consensus.cc:482] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:32.198150 18572 raft_consensus.cc:3037] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:32.202422 18572 raft_consensus.cc:504] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.203308 18572 leader_election.cc:288] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers 00e1b972f516418c98271614df46576a (127.0.97.3:39583), c7eb06ab221a4842bb864d5f2e18e9bf (127.0.97.1:43607)
I20260927 15:11:32.203924 18500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df5e995a2645b4905a2977170cd735" candidate_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e1b972f516418c98271614df46576a"
I20260927 15:11:32.204177 18500 raft_consensus.cc:3037] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:32.204363 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df5e995a2645b4905a2977170cd735" candidate_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf"
I20260927 15:11:32.204650 18238 raft_consensus.cc:3037] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 0 FOLLOWER]: Advancing to term 1
W20260927 15:11:32.206204 18546 tablet.cc:2369] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:32.207059 18238 raft_consensus.cc:2454] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc397e59b1b408f9ffcaa90f19c2b5f in term 1.
I20260927 15:11:32.207602 18305 leader_election.cc:302] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc397e59b1b408f9ffcaa90f19c2b5f, c7eb06ab221a4842bb864d5f2e18e9bf; no voters:
I20260927 15:11:32.207638 18500 raft_consensus.cc:2454] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc397e59b1b408f9ffcaa90f19c2b5f in term 1.
I20260927 15:11:32.207911 18572 raft_consensus.cc:2781] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:32.208480 18572 raft_consensus.cc:686] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 3bc397e59b1b408f9ffcaa90f19c2b5f, State: Running, Role: LEADER
I20260927 15:11:32.208984 18572 consensus_queue.cc:237] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } }
I20260927 15:11:32.214473 18097 catalog_manager.cc:5495] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bc397e59b1b408f9ffcaa90f19c2b5f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc397e59b1b408f9ffcaa90f19c2b5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:32.253947 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.248756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.97.2:40421)
0927 15:11:32.248924 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.248929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.248946 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:11:32.249099 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.249103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.249194 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.249327 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:32.249707 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.250363 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.251920 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.252336 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.252433 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.252897 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:32.253394 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:32.253505 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.253634 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.253733 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:32.262367 18500 raft_consensus.cc:1261] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 1 FOLLOWER]: Refusing update from remote peer 3bc397e59b1b408f9ffcaa90f19c2b5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:32.262367 18238 raft_consensus.cc:1261] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 1 FOLLOWER]: Refusing update from remote peer 3bc397e59b1b408f9ffcaa90f19c2b5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:32.263506 18575 consensus_queue.cc:1035] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7eb06ab221a4842bb864d5f2e18e9bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:32.263852 18572 consensus_queue.cc:1035] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1b972f516418c98271614df46576a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:32.291919 18096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:32.292263 18096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:32.296854 18096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c0a12f3edfc47e1815095eea18d40c9: Sending DeleteTablet for 3 replicas of tablet 46df5e995a2645b4905a2977170cd735
I20260927 15:11:32.298198 18349 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df5e995a2645b4905a2977170cd735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:32 UTC) from {username='slave'} at 127.0.0.1:47574
I20260927 15:11:32.298930 18480 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df5e995a2645b4905a2977170cd735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:32 UTC) from {username='slave'} at 127.0.0.1:44914
I20260927 15:11:32.299306 18588 tablet_replica.cc:330] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: stopping tablet replica
I20260927 15:11:32.299816 18588 raft_consensus.cc:2229] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:32.300570 18588 raft_consensus.cc:2258] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:32.302675 18218 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df5e995a2645b4905a2977170cd735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:32 UTC) from {username='slave'} at 127.0.0.1:55608
I20260927 15:11:32.302927 18588 ts_tablet_manager.cc:1905] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:32.310722 18588 ts_tablet_manager.cc:1918] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:32.311110 18588 log.cc:1196] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/46df5e995a2645b4905a2977170cd735
I20260927 15:11:32.311474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18156
I20260927 15:11:32.311882 18588 ts_tablet_manager.cc:1939] T 46df5e995a2645b4905a2977170cd735 P 3bc397e59b1b408f9ffcaa90f19c2b5f: Deleting consensus metadata
I20260927 15:11:32.313932 18082 catalog_manager.cc:4841] TS 3bc397e59b1b408f9ffcaa90f19c2b5f (127.0.97.2:40421): tablet 46df5e995a2645b4905a2977170cd735 (table test-table [id=fc7bce703bf04f1bbc421bee5c08283e]) successfully deleted
I20260927 15:11:32.319782 18590 tablet_replica.cc:330] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: stopping tablet replica
I20260927 15:11:32.320451 18590 raft_consensus.cc:2229] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:32.321118 18590 raft_consensus.cc:2258] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:32.321811 18589 tablet_replica.cc:330] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: stopping tablet replica
I20260927 15:11:32.322530 18589 raft_consensus.cc:2229] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:32.323503 18590 ts_tablet_manager.cc:1905] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:32.324062 18589 raft_consensus.cc:2258] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:32.326596 18589 ts_tablet_manager.cc:1905] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:32.331413 18590 ts_tablet_manager.cc:1918] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:32.331633 18590 log.cc:1196] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/46df5e995a2645b4905a2977170cd735
I20260927 15:11:32.332299 18590 ts_tablet_manager.cc:1939] T 46df5e995a2645b4905a2977170cd735 P c7eb06ab221a4842bb864d5f2e18e9bf: Deleting consensus metadata
I20260927 15:11:32.334473 18084 catalog_manager.cc:4841] TS c7eb06ab221a4842bb864d5f2e18e9bf (127.0.97.1:43607): tablet 46df5e995a2645b4905a2977170cd735 (table test-table [id=fc7bce703bf04f1bbc421bee5c08283e]) successfully deleted
I20260927 15:11:32.336609 18589 ts_tablet_manager.cc:1918] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:32.336874 18589 log.cc:1196] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/46df5e995a2645b4905a2977170cd735
I20260927 15:11:32.337513 18589 ts_tablet_manager.cc:1939] T 46df5e995a2645b4905a2977170cd735 P 00e1b972f516418c98271614df46576a: Deleting consensus metadata
I20260927 15:11:32.339586 18081 catalog_manager.cc:4841] TS 00e1b972f516418c98271614df46576a (127.0.97.3:39583): tablet 46df5e995a2645b4905a2977170cd735 (table test-table [id=fc7bce703bf04f1bbc421bee5c08283e]) successfully deleted
I20260927 15:11:32.339798 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.313037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.0.97.1:43607)
0927 15:11:32.313269 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.313278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.331694 (+ 18416us) server_negotiation.cc:407] Connection header received
0927 15:11:32.332062 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.332068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.332177 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.332266 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:32.333036 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.334161 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.336219 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.336853 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.336989 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.338289 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:32.338299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:32.338312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:32.338911 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:32.338968 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:32.338989 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:32.339166 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.339291 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.339416 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:11:32.340637 18278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:32.467470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18156
I20260927 15:11:32.486285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18287
I20260927 15:11:32.495296 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.487404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.0.97.2:40421)
0927 15:11:32.487592 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.487597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.487614 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:11:32.487680 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.487684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.487766 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.487853 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:32.488418 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.489301 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.491519 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.492034 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.492161 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.493573 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:32.493582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:32.493598 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:32.494329 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:32.494415 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:32.494439 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:32.494544 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.494737 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.494842 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:11:32.496129 18409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:32.612816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18287
I20260927 15:11:32.632048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18418
I20260927 15:11:32.639360 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.633011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.0.97.3:39583)
0927 15:11:32.633224 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.633230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.633252 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:11:32.633442 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.633446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.633546 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.633656 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:32.634061 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.634787 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.636167 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.636571 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.636708 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.637855 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:32.637865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:32.637880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:32.638504 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:32.638555 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:32.638577 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:32.638735 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.638869 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.638990 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:11:32.640098 18540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:32.730640 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18418
I20260927 15:11:32.748380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18066
I20260927 15:11:32.749519 18127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:32.861325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18066
2026-09-27T15:11:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3116 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:32 dist-test-slave-g0w1 krb5kdc[18598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:32 dist-test-slave-g0w1 krb5kdc[18598](info): set up 2 sockets
Sep 27 15:11:32 dist-test-slave-g0w1 krb5kdc[18598](info): commencing operation
krb5kdc: starting...
W20260927 15:11:34.003367 388 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:34 dist-test-slave-g0w1 krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:11:34.266880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43827
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:34.637387 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:34.637727 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:34.637900 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:34.645465 18611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:34.645596 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:34.645694 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:34.645816 18611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:34.645903 18611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:34.656236 18611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:34.657737 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:34.659214 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:34.672274 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:34.672253 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:34.672408 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:34.673717 18611 server_base.cc:1034] running on GCE node
I20260927 15:11:34.674723 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:34.676432 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:34.677708 18611 hybrid_clock.cc:648] HybridClock initialized: now 1790521894677662 us; error 45 us; skew 500 ppm
I20260927 15:11:34.678133 18611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:34 dist-test-slave-g0w1 krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:34.681788 18611 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:11:34.683640 18611 webserver.cc:460] Webserver started at http://127.0.97.62:38381/ using document root <none> and password file <none>
I20260927 15:11:34.684424 18611 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:34.684597 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:34.684939 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:34.688405 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "4c5003549a704bf19a7254127a996fe6"
format_stamp: "Formatted at 2026-09-27 15:11:34 on dist-test-slave-g0w1"
server_key: "b78d0603ce1ea01f3acd147cf0bab3db"
server_key_iv: "03349c0d39b14a9cdec0a0b0ad6f08eb"
server_key_version: "encryptionkey@0"
I20260927 15:11:34.689299 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c5003549a704bf19a7254127a996fe6"
format_stamp: "Formatted at 2026-09-27 15:11:34 on dist-test-slave-g0w1"
server_key: "b78d0603ce1ea01f3acd147cf0bab3db"
server_key_iv: "03349c0d39b14a9cdec0a0b0ad6f08eb"
server_key_version: "encryptionkey@0"
I20260927 15:11:34.695573 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:11:34.699826 18626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:34.701606 18611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:34.701861 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "4c5003549a704bf19a7254127a996fe6"
format_stamp: "Formatted at 2026-09-27 15:11:34 on dist-test-slave-g0w1"
server_key: "b78d0603ce1ea01f3acd147cf0bab3db"
server_key_iv: "03349c0d39b14a9cdec0a0b0ad6f08eb"
server_key_version: "encryptionkey@0"
I20260927 15:11:34.702119 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:34.728227 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:34.733359 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:34.733688 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:34.753880 18611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43827
I20260927 15:11:34.753904 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43827 every 8 connection(s)
I20260927 15:11:34.755455 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:34.759605 18678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:34.763913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18611
I20260927 15:11:34.764149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:34.764608 388 external_mini_cluster.cc:1428] Setting key 9da72c29e4348a3510e73e56da9099f1
I20260927 15:11:34.772895 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6: Bootstrap starting.
I20260927 15:11:34.776949 18678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:34.778347 18678 log.cc:824] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:34.781952 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6: No bootstrap required, opened a new log
Sep 27 15:11:34 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:34.792230 18678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5003549a704bf19a7254127a996fe6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43827 } }
I20260927 15:11:34.792685 18678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:34.792821 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:34.766659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.97.62:43827)
0927 15:11:34.767577 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:34.767596 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:34.767656 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:11:34.770895 (+ 3239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:34.770930 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:34.771562 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:34.772075 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:34.773795 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:34.775960 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:34.777531 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:34.778710 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:34.778947 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:34.783761 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:34.783771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:34.783789 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:34.783837 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:34.787385 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:34.788481 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:34.788490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:34.788498 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:34.788716 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:34.789226 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:34.789232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:34.789235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:34.790003 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:34.790268 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:34.790797 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:11:34.791472 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":407,"threads_started":1}
I20260927 15:11:34.792843 18678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5003549a704bf19a7254127a996fe6, State: Initialized, Role: FOLLOWER
I20260927 15:11:34.793637 18678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5003549a704bf19a7254127a996fe6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43827 } }
I20260927 15:11:34.793927 18678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:34.794087 18678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:34.794286 18678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:34.797206 18678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5003549a704bf19a7254127a996fe6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43827 } }
I20260927 15:11:34.797840 18678 leader_election.cc:302] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c5003549a704bf19a7254127a996fe6; no voters:
I20260927 15:11:34.798496 18678 leader_election.cc:288] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:34.798661 18683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:34.799463 18683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [term 1 LEADER]: Becoming Leader. State: Replica: 4c5003549a704bf19a7254127a996fe6, State: Running, Role: LEADER
I20260927 15:11:34.800093 18683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5003549a704bf19a7254127a996fe6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43827 } }
I20260927 15:11:34.800863 18678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:34.802803 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c5003549a704bf19a7254127a996fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5003549a704bf19a7254127a996fe6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43827 } } }
I20260927 15:11:34.803254 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:34.803555 18685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5003549a704bf19a7254127a996fe6. Latest consensus state: current_term: 1 leader_uuid: "4c5003549a704bf19a7254127a996fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5003549a704bf19a7254127a996fe6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43827 } } }
I20260927 15:11:34.803944 18685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:34.806602 18688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:34.812687 18688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:34.822748 18688 catalog_manager.cc:1347] Generated new cluster ID: 44cef976732a4393a96c352b7b3b842b
I20260927 15:11:34.822926 18688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:34.843547 18688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:34.845729 18688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:34.862200 18688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6: Generated new TSK 0
I20260927 15:11:34.863518 18688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:11:34.931118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43827
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:35.294080 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:35.294392 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:35.294559 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:35.302569 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:35.302695 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:35.302901 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:35.314389 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:35.315955 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:35.317454 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:35.332237 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:35.332563 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:35.332839 18704 server_base.cc:1034] running on GCE node
W20260927 15:11:35.332254 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:35.333730 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:35.334736 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:35.335944 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790521895335878 us; error 106 us; skew 500 ppm
I20260927 15:11:35.336251 18704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:35 dist-test-slave-g0w1 krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:35.346271 18704 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:11:35.348057 18704 webserver.cc:460] Webserver started at http://127.0.97.1:38967/ using document root <none> and password file <none>
I20260927 15:11:35.348856 18704 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:35.349031 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:35.349373 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:35.352646 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "dc4fb806371349c5a453cf8af2ab8c1d"
format_stamp: "Formatted at 2026-09-27 15:11:35 on dist-test-slave-g0w1"
server_key: "de17320b6c75f85b72741cdc00939b72"
server_key_iv: "d9b5267a7a0ee58fe85b675f0959ce2e"
server_key_version: "encryptionkey@0"
I20260927 15:11:35.353493 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dc4fb806371349c5a453cf8af2ab8c1d"
format_stamp: "Formatted at 2026-09-27 15:11:35 on dist-test-slave-g0w1"
server_key: "de17320b6c75f85b72741cdc00939b72"
server_key_iv: "d9b5267a7a0ee58fe85b675f0959ce2e"
server_key_version: "encryptionkey@0"
I20260927 15:11:35.359382 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:11:35.363927 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:35.365679 18704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:35.365940 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "dc4fb806371349c5a453cf8af2ab8c1d"
format_stamp: "Formatted at 2026-09-27 15:11:35 on dist-test-slave-g0w1"
server_key: "de17320b6c75f85b72741cdc00939b72"
server_key_iv: "d9b5267a7a0ee58fe85b675f0959ce2e"
server_key_version: "encryptionkey@0"
I20260927 15:11:35.366189 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:35.384843 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:35.389715 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:35.390044 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:35.391215 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:35.392735 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:35.392853 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:35.393018 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:35.393103 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:35.429730 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43281
I20260927 15:11:35.429797 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43281 every 8 connection(s)
I20260927 15:11:35.431190 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:35.438055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18704
I20260927 15:11:35.438261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:35.438843 388 external_mini_cluster.cc:1428] Setting key f43d1821465fd271585e36f62ab9b158
Sep 27 15:11:35 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:35.453572 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:35.434149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37579 (local address 127.0.97.62:43827)
0927 15:11:35.434635 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:35.434640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:35.435803 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 15:11:35.437691 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:35.437696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:35.437769 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:35.437873 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:35.440030 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:35.440804 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:35.442669 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:35.443182 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:35.443331 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:35.447154 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:35.447160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:35.447163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:35.447199 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:35.450037 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:35.450934 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:35.450943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:35.450950 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:35.451139 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:35.451629 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:35.451636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:35.451639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:35.451938 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:35.452118 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:35.452930 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 15:11:35.453192 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":231,"threads_started":1}
I20260927 15:11:35.454651 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:35.434755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43827 (local address 127.0.97.1:37579)
0927 15:11:35.435608 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:35.435655 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:35.437378 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:35.438198 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:35.438211 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:35.439038 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:35.439821 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:35.439842 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:35.441024 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:35.441039 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:35.442476 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:35.442492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:35.443452 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:35.443460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:35.443716 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:35.444470 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:35.444494 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:35.446854 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:35.450218 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:35.450228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:35.450247 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:35.450745 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:35.451259 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:35.451264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:35.451268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:35.451501 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:35.452090 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:35.452099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:35.452524 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 15:11:35.452812 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 15:11:35.453414 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":370,"threads_started":1}
I20260927 15:11:35.456609 18832 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43827
I20260927 15:11:35.457021 18832 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:35.458137 18832 heartbeater.cc:511] Master 127.0.97.62:43827 requested a full tablet report, sending...
I20260927 15:11:35.460891 18643 ts_manager.cc:194] Registered new tserver with Master: dc4fb806371349c5a453cf8af2ab8c1d (127.0.97.1:43281)
I20260927 15:11:35.462512 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37579
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:11:35.502858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43827
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:35.865387 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:35.865875 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:35.866161 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:35.882261 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:35.882476 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:35.882772 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:35.901448 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:35.903190 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:35.904737 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:35.920785 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:35.920732 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:35.920712 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:35.921761 18838 server_base.cc:1034] running on GCE node
I20260927 15:11:35.922523 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:35.923753 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:35.925078 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790521895924995 us; error 91 us; skew 500 ppm
I20260927 15:11:35.925484 18838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:35 dist-test-slave-g0w1 krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:35.929582 18838 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:11:35.931506 18838 webserver.cc:460] Webserver started at http://127.0.97.2:41941/ using document root <none> and password file <none>
I20260927 15:11:35.932263 18838 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:35.932389 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:35.932673 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:35.936573 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "47ec8925c2694970a89ef96982266211"
format_stamp: "Formatted at 2026-09-27 15:11:35 on dist-test-slave-g0w1"
server_key: "a438bca9982976868bc830629200c6fb"
server_key_iv: "08c289464f19389454c67033f9dd864e"
server_key_version: "encryptionkey@0"
I20260927 15:11:35.937798 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "47ec8925c2694970a89ef96982266211"
format_stamp: "Formatted at 2026-09-27 15:11:35 on dist-test-slave-g0w1"
server_key: "a438bca9982976868bc830629200c6fb"
server_key_iv: "08c289464f19389454c67033f9dd864e"
server_key_version: "encryptionkey@0"
I20260927 15:11:35.948228 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20260927 15:11:35.954064 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:35.957614 18838 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 15:11:35.957962 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "47ec8925c2694970a89ef96982266211"
format_stamp: "Formatted at 2026-09-27 15:11:35 on dist-test-slave-g0w1"
server_key: "a438bca9982976868bc830629200c6fb"
server_key_iv: "08c289464f19389454c67033f9dd864e"
server_key_version: "encryptionkey@0"
I20260927 15:11:35.958613 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:35.974855 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:35.980381 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:35.980746 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:35.981802 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:35.983584 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:35.983680 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:35.983774 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:35.983819 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:36.025214 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33381
I20260927 15:11:36.025264 18965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33381 every 8 connection(s)
I20260927 15:11:36.026629 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:36.031886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18838
I20260927 15:11:36.032099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:36.032562 388 external_mini_cluster.cc:1428] Setting key 8e129683b2035caca1e21a48b82aecd1
Sep 27 15:11:36 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:36.050559 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.029550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47591 (local address 127.0.97.62:43827)
0927 15:11:36.030025 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.030030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.031130 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 15:11:36.033447 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.033452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.033549 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.033685 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:36.036380 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.037142 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.038886 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.039418 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.039695 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.043814 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:36.043820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:36.043824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:36.043862 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:36.046959 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.047948 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.047953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.047955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.048104 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.048608 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.048611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.048613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.048905 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:36.049094 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.049881 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.050178 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260927 15:11:36.051688 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.030161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43827 (local address 127.0.97.2:47591)
0927 15:11:36.030914 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:36.030962 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:36.032998 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:36.034031 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:36.034044 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:36.034967 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:36.036128 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.036152 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.037325 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.037337 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.038693 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.038710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.039792 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.039803 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.040140 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.041012 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:36.041038 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:36.043516 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:36.047207 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:36.047220 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:36.047240 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:36.047772 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:36.048233 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:36.048239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:36.048242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:36.048463 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:36.049056 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:36.049065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:36.049467 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 15:11:36.049762 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 15:11:36.050371 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260927 15:11:36.053503 18966 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43827
I20260927 15:11:36.053894 18966 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:36.054687 18966 heartbeater.cc:511] Master 127.0.97.62:43827 requested a full tablet report, sending...
I20260927 15:11:36.056535 18643 ts_manager.cc:194] Registered new tserver with Master: 47ec8925c2694970a89ef96982266211 (127.0.97.2:33381)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:11:36.057451 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:11:36.091022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43827
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:36.448858 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:36.449178 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:36.449275 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:36.457121 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:36.457265 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:36.457388 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:36.466842 18832 heartbeater.cc:503] Master 127.0.97.62:43827 was elected leader, sending a full tablet report...
I20260927 15:11:36.468899 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:36.470544 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:36.472023 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:36.486826 18978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:36.486917 18980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:36.487213 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:36.488094 18972 server_base.cc:1034] running on GCE node
I20260927 15:11:36.488812 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:36.489997 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:36.491187 18972 hybrid_clock.cc:648] HybridClock initialized: now 1790521896491143 us; error 39 us; skew 500 ppm
I20260927 15:11:36.491564 18972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:36 dist-test-slave-g0w1 krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:36.495030 18972 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:11:36.496865 18972 webserver.cc:460] Webserver started at http://127.0.97.3:40559/ using document root <none> and password file <none>
I20260927 15:11:36.497639 18972 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:36.497751 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:36.498126 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:36.501395 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "2f6485b55cd74fe0be04c06fc08ab187"
format_stamp: "Formatted at 2026-09-27 15:11:36 on dist-test-slave-g0w1"
server_key: "9e51224f86dc9379480e41cd646c8ffa"
server_key_iv: "0f0173d6c6695f5b30603661f80fe730"
server_key_version: "encryptionkey@0"
I20260927 15:11:36.502328 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2f6485b55cd74fe0be04c06fc08ab187"
format_stamp: "Formatted at 2026-09-27 15:11:36 on dist-test-slave-g0w1"
server_key: "9e51224f86dc9379480e41cd646c8ffa"
server_key_iv: "0f0173d6c6695f5b30603661f80fe730"
server_key_version: "encryptionkey@0"
I20260927 15:11:36.508324 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:11:36.512468 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:36.513926 18972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:36.514142 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "2f6485b55cd74fe0be04c06fc08ab187"
format_stamp: "Formatted at 2026-09-27 15:11:36 on dist-test-slave-g0w1"
server_key: "9e51224f86dc9379480e41cd646c8ffa"
server_key_iv: "0f0173d6c6695f5b30603661f80fe730"
server_key_version: "encryptionkey@0"
I20260927 15:11:36.514372 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:36.546948 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:36.552104 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:36.552438 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:36.553505 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:36.555028 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:36.555163 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:11:36.555325 18972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:36.555413 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:36.592021 18972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46629
I20260927 15:11:36.592080 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46629 every 8 connection(s)
I20260927 15:11:36.593453 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:36.598551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 18972
I20260927 15:11:36.598753 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:36.599349 388 external_mini_cluster.cc:1428] Setting key b47b0865acf6b95362246be74e46a5d0
Sep 27 15:11:36 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:36.616333 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.596459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60915 (local address 127.0.97.62:43827)
0927 15:11:36.596994 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.596999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.597886 (+ 887us) server_negotiation.cc:407] Connection header received
0927 15:11:36.600052 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.600059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.600159 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.600288 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:36.602163 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.603137 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.604797 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.605237 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.605377 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.609499 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:36.609505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:36.609513 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:36.609551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:36.612541 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.613597 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.613605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.613610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.613765 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.614286 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.614291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.614293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.614709 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:36.614904 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.615801 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.616020 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260927 15:11:36.617400 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.596907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43827 (local address 127.0.97.3:60915)
0927 15:11:36.597708 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:36.597750 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:36.599751 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:36.600561 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:36.600571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:36.601383 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:36.601989 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.602008 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.603310 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.603314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.604620 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.604633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.605483 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.605491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.605737 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.606527 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:36.606567 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:36.609236 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:36.612789 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:36.612799 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:36.612821 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:36.613447 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:36.613890 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:36.613895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:36.613900 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:36.614160 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:36.614897 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:36.614905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:36.615359 (+ 454us) client_negotiation.cc:769] Sending connection context
0927 15:11:36.615701 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 15:11:36.616177 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":301,"threads_started":1}
I20260927 15:11:36.619020 19100 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43827
I20260927 15:11:36.619408 19100 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:36.620236 19100 heartbeater.cc:511] Master 127.0.97.62:43827 requested a full tablet report, sending...
I20260927 15:11:36.621766 18643 ts_manager.cc:194] Registered new tserver with Master: 2f6485b55cd74fe0be04c06fc08ab187 (127.0.97.3:46629)
I20260927 15:11:36.622660 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60915
I20260927 15:11:36.628173 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:36.645865 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.632465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.0.97.62:43827)
0927 15:11:36.632643 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.632649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.633199 (+ 550us) server_negotiation.cc:407] Connection header received
0927 15:11:36.633326 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.633330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.633412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.633512 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:36.635022 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.635876 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.637292 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.637725 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.637830 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.640613 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:36.640618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:36.640621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:36.640657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:36.643341 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.644118 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.644123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.644125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.644245 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.644628 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.644631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.644633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.644929 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:36.645124 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.645476 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.645622 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:11:36.650305 18643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:36.657402 18643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:36.683745 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.670090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46629 (local address 127.0.0.1:36756)
0927 15:11:36.673101 (+ 3011us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:36.673139 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:36.673339 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:36.675229 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:36.675234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:36.675265 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:36.675664 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.675675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.677682 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.677687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.680540 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.680557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.682497 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.682503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.682723 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.682784 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 15:11:36.682925 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 15:11:36.683185 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2763,"mutex_wait_us":105,"thread_start_us":2624,"threads_started":1}
I20260927 15:11:36.683772 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.673882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.0.97.2:33381)
0927 15:11:36.674497 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.674508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.675236 (+ 728us) server_negotiation.cc:407] Connection header received
0927 15:11:36.675539 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.675550 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.675798 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.675982 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:36.676819 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.678269 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.680884 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.682316 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.682445 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.682661 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.683261 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.683440 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":318,"threads_started":1}
I20260927 15:11:36.684674 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.674528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33381 (local address 127.0.0.1:46658)
0927 15:11:36.675136 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:36.675157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:36.675330 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:36.676225 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:36.676229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:36.676255 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:36.676616 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.676632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.678430 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.678436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.680722 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.680734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.682821 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.682828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.683024 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.683056 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:11:36.684186 (+ 1130us) client_negotiation.cc:240] Negotiation successful
0927 15:11:36.684337 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":291,"threads_started":1}
I20260927 15:11:36.684791 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.670348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.0.97.3:46629)
0927 15:11:36.671418 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.671444 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.673285 (+ 1841us) server_negotiation.cc:407] Connection header received
0927 15:11:36.673615 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.673627 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.673910 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.675368 (+ 1458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:36.675833 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.677525 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.680725 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.682289 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.683941 (+ 1652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.684201 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.684275 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.684466 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":1266,"threads_started":1}
I20260927 15:11:36.684978 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.671040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.97.1:43281)
0927 15:11:36.672503 (+ 1463us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.672516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.674050 (+ 1534us) server_negotiation.cc:407] Connection header received
0927 15:11:36.674285 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.674294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.674544 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.674737 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:36.676179 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.677515 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.681999 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.683453 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.683585 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.683843 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.684480 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.684635 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1004,"threads_started":1}
I20260927 15:11:36.685753 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.670391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43281 (local address 127.0.0.1:53930)
0927 15:11:36.673900 (+ 3509us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:36.673917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:36.674057 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:36.675158 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:36.675167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:36.675200 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:36.676005 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.676012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.677657 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.677664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.681819 (+ 4155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:36.681831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.683970 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.683975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:36.684249 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.684281 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:11:36.685230 (+ 949us) client_negotiation.cc:240] Negotiation successful
0927 15:11:36.685388 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3397,"mutex_wait_us":30,"thread_start_us":3252,"threads_started":1}
I20260927 15:11:36.688752 18901 tablet_service.cc:1467] Processing CreateTablet for tablet 46a3751df5f846c8859e5b40bf822a96 (DEFAULT_TABLE table=test-table [id=f49f56df20594119a6dfedd81a89a95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:36.688751 19035 tablet_service.cc:1467] Processing CreateTablet for tablet 46a3751df5f846c8859e5b40bf822a96 (DEFAULT_TABLE table=test-table [id=f49f56df20594119a6dfedd81a89a95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:36.689203 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 46a3751df5f846c8859e5b40bf822a96 (DEFAULT_TABLE table=test-table [id=f49f56df20594119a6dfedd81a89a95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:36.690132 18901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a3751df5f846c8859e5b40bf822a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:36.690207 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a3751df5f846c8859e5b40bf822a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:36.690467 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a3751df5f846c8859e5b40bf822a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:36.703508 19119 tablet_bootstrap.cc:492] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Bootstrap starting.
I20260927 15:11:36.705291 19120 tablet_bootstrap.cc:492] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Bootstrap starting.
I20260927 15:11:36.706138 19121 tablet_bootstrap.cc:492] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Bootstrap starting.
I20260927 15:11:36.707413 19119 tablet_bootstrap.cc:654] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:36.708901 19119 log.cc:824] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:36.710058 19121 tablet_bootstrap.cc:654] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:36.710418 19120 tablet_bootstrap.cc:654] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:36.711496 19121 log.cc:824] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:36.711798 19120 log.cc:824] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:36.713652 19119 tablet_bootstrap.cc:492] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: No bootstrap required, opened a new log
I20260927 15:11:36.713968 19119 ts_tablet_manager.cc:1397] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 15:11:36.714821 19121 tablet_bootstrap.cc:492] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: No bootstrap required, opened a new log
I20260927 15:11:36.715149 19120 tablet_bootstrap.cc:492] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: No bootstrap required, opened a new log
I20260927 15:11:36.715154 19121 ts_tablet_manager.cc:1397] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 15:11:36.715416 19120 ts_tablet_manager.cc:1397] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 15:11:36.722316 19119 raft_consensus.cc:348] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:36.722894 19119 raft_consensus.cc:374] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:36.723006 19119 raft_consensus.cc:729] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ec8925c2694970a89ef96982266211, State: Initialized, Role: FOLLOWER
I20260927 15:11:36.723924 19119 consensus_queue.cc:260] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:36.725200 18966 heartbeater.cc:503] Master 127.0.97.62:43827 was elected leader, sending a full tablet report...
I20260927 15:11:36.726574 19120 raft_consensus.cc:348] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:36.726574 19121 raft_consensus.cc:348] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:36.727054 19121 raft_consensus.cc:374] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:36.727054 19120 raft_consensus.cc:374] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:36.727187 19121 raft_consensus.cc:729] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6485b55cd74fe0be04c06fc08ab187, State: Initialized, Role: FOLLOWER
I20260927 15:11:36.727187 19120 raft_consensus.cc:729] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4fb806371349c5a453cf8af2ab8c1d, State: Initialized, Role: FOLLOWER
I20260927 15:11:36.727365 19119 ts_tablet_manager.cc:1428] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260927 15:11:36.727856 19121 consensus_queue.cc:260] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:36.727856 19120 consensus_queue.cc:260] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:36.729687 19100 heartbeater.cc:503] Master 127.0.97.62:43827 was elected leader, sending a full tablet report...
I20260927 15:11:36.729945 19121 ts_tablet_manager.cc:1428] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260927 15:11:36.730214 19120 ts_tablet_manager.cc:1428] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
W20260927 15:11:36.779006 18967 tablet.cc:2369] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:36.845327 19101 tablet.cc:2369] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:36.933979 18833 tablet.cc:2369] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:36.998860 19125 raft_consensus.cc:482] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:36.999204 19125 raft_consensus.cc:504] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:37.000837 19126 raft_consensus.cc:482] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:37.001067 19125 leader_election.cc:288] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6485b55cd74fe0be04c06fc08ab187 (127.0.97.3:46629), dc4fb806371349c5a453cf8af2ab8c1d (127.0.97.1:43281)
I20260927 15:11:37.001189 19126 raft_consensus.cc:504] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:37.006011 19126 leader_election.cc:288] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc4fb806371349c5a453cf8af2ab8c1d (127.0.97.1:43281), 47ec8925c2694970a89ef96982266211 (127.0.97.2:33381)
I20260927 15:11:37.011420 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.001765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38577 (local address 127.0.97.3:46629)
0927 15:11:37.001999 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.002007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.002615 (+ 608us) server_negotiation.cc:407] Connection header received
0927 15:11:37.002934 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.002940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.003024 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.003180 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:37.003823 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.005055 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.008638 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.010022 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.010126 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.010246 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.010925 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.011100 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 15:11:37.011620 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.001650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49577 (local address 127.0.97.1:43281)
0927 15:11:37.001901 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.001909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.002853 (+ 944us) server_negotiation.cc:407] Connection header received
0927 15:11:37.003123 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.003129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.003209 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.003294 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:37.004136 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.005215 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.009110 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.010379 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.010491 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.010591 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.011213 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.011366 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:11:37.011942 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.003544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33381 (local address 127.0.97.3:40761)
0927 15:11:37.003949 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:37.003968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:37.004071 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:37.004803 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:37.004811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:37.004846 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:37.005187 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.005202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.007022 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.007030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.009205 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.009220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.011140 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.011146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.011363 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.011396 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:11:37.011486 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:11:37.011654 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260927 15:11:37.012396 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.001324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46629 (local address 127.0.97.2:38577)
0927 15:11:37.002473 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:37.002502 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:37.002740 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:37.003201 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:37.003205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:37.003229 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:37.003626 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.003643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.005244 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.005252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.008440 (+ 3188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.008458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.010470 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.010477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.010670 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.010717 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 15:11:37.011724 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0927 15:11:37.011901 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":2,"thread_start_us":1142,"threads_started":1}
I20260927 15:11:37.012476 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.001471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43281 (local address 127.0.97.2:49577)
0927 15:11:37.002764 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:37.002781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:37.002942 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:37.003466 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:37.003472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:37.003502 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:37.003943 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.003957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.005384 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.005388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.008920 (+ 3532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.008935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.010794 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.010803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.011009 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.011043 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:11:37.011842 (+ 799us) client_negotiation.cc:240] Negotiation successful
0927 15:11:37.011954 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":784,"thread_start_us":607,"threads_started":1}
I20260927 15:11:37.013322 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.004096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40761 (local address 127.0.97.2:33381)
0927 15:11:37.004299 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.004305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.004324 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:11:37.004386 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.004392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.004458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.004548 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:37.005418 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.006796 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.009435 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.010747 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.010923 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.011013 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.012235 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.012418 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260927 15:11:37.013444 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "47ec8925c2694970a89ef96982266211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6485b55cd74fe0be04c06fc08ab187" is_pre_election: true
I20260927 15:11:37.013581 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "47ec8925c2694970a89ef96982266211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" is_pre_election: true
I20260927 15:11:37.013922 19055 raft_consensus.cc:2454] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47ec8925c2694970a89ef96982266211 in term 0.
I20260927 15:11:37.014093 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "2f6485b55cd74fe0be04c06fc08ab187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ec8925c2694970a89ef96982266211" is_pre_election: true
I20260927 15:11:37.014189 18787 raft_consensus.cc:2454] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47ec8925c2694970a89ef96982266211 in term 0.
I20260927 15:11:37.014530 18921 raft_consensus.cc:2454] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6485b55cd74fe0be04c06fc08ab187 in term 0.
I20260927 15:11:37.014894 18856 leader_election.cc:302] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6485b55cd74fe0be04c06fc08ab187, 47ec8925c2694970a89ef96982266211; no voters:
I20260927 15:11:37.015447 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.003544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43281 (local address 127.0.97.3:36407)
0927 15:11:37.005891 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:37.005914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:37.006085 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:37.007211 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:37.007215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:37.007241 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:37.007661 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.007672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.009511 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.009518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.012382 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:37.012396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.014559 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.014567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:37.014803 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.014837 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:11:37.014898 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 15:11:37.015051 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2211,"mutex_wait_us":94,"thread_start_us":1961,"threads_started":1}
I20260927 15:11:37.015465 19125 raft_consensus.cc:2781] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:37.015612 19125 raft_consensus.cc:482] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:37.015717 19125 raft_consensus.cc:3037] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:37.015865 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.004420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36407 (local address 127.0.97.1:43281)
0927 15:11:37.006796 (+ 2376us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.006803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.006825 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:11:37.006895 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.006899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.006970 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.007831 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:37.008120 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.009322 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.012842 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.014327 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.015191 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.015314 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.015379 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.015513 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2246,"thread_start_us":1988,"threads_started":1}
I20260927 15:11:37.015842 18990 leader_election.cc:302] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6485b55cd74fe0be04c06fc08ab187, 47ec8925c2694970a89ef96982266211; no voters:
I20260927 15:11:37.016294 19126 raft_consensus.cc:2781] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:37.016438 19126 raft_consensus.cc:482] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:37.016542 19126 raft_consensus.cc:3037] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:37.016431 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "2f6485b55cd74fe0be04c06fc08ab187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" is_pre_election: true
I20260927 15:11:37.016685 18787 raft_consensus.cc:2454] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6485b55cd74fe0be04c06fc08ab187 in term 0.
I20260927 15:11:37.019021 19125 raft_consensus.cc:504] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:37.019749 19126 raft_consensus.cc:504] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:37.020278 19125 leader_election.cc:288] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6485b55cd74fe0be04c06fc08ab187 (127.0.97.3:46629), dc4fb806371349c5a453cf8af2ab8c1d (127.0.97.1:43281)
I20260927 15:11:37.020496 19126 leader_election.cc:288] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [CANDIDATE]: Term 1 election: Requested vote from peers dc4fb806371349c5a453cf8af2ab8c1d (127.0.97.1:43281), 47ec8925c2694970a89ef96982266211 (127.0.97.2:33381)
I20260927 15:11:37.020974 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "47ec8925c2694970a89ef96982266211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4fb806371349c5a453cf8af2ab8c1d"
I20260927 15:11:37.021215 18787 raft_consensus.cc:3037] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:37.021162 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "2f6485b55cd74fe0be04c06fc08ab187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ec8925c2694970a89ef96982266211"
I20260927 15:11:37.021934 18786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "2f6485b55cd74fe0be04c06fc08ab187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4fb806371349c5a453cf8af2ab8c1d"
I20260927 15:11:37.022167 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a3751df5f846c8859e5b40bf822a96" candidate_uuid: "47ec8925c2694970a89ef96982266211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6485b55cd74fe0be04c06fc08ab187"
I20260927 15:11:37.022451 19055 raft_consensus.cc:2379] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 47ec8925c2694970a89ef96982266211 in current term 1: Already voted for candidate 2f6485b55cd74fe0be04c06fc08ab187 in this term.
I20260927 15:11:37.022899 18921 raft_consensus.cc:2379] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f6485b55cd74fe0be04c06fc08ab187 in current term 1: Already voted for candidate 47ec8925c2694970a89ef96982266211 in this term.
I20260927 15:11:37.023586 18990 leader_election.cc:302] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f6485b55cd74fe0be04c06fc08ab187; no voters: 47ec8925c2694970a89ef96982266211, dc4fb806371349c5a453cf8af2ab8c1d
I20260927 15:11:37.023882 19126 raft_consensus.cc:2726] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 15:11:37.024585 18787 raft_consensus.cc:2454] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47ec8925c2694970a89ef96982266211 in term 1.
I20260927 15:11:37.025058 18855 leader_election.cc:302] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47ec8925c2694970a89ef96982266211, dc4fb806371349c5a453cf8af2ab8c1d; no voters: 2f6485b55cd74fe0be04c06fc08ab187
I20260927 15:11:37.025588 19125 raft_consensus.cc:2781] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:37.026063 19125 raft_consensus.cc:686] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 1 LEADER]: Becoming Leader. State: Replica: 47ec8925c2694970a89ef96982266211, State: Running, Role: LEADER
I20260927 15:11:37.026616 19125 consensus_queue.cc:237] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } }
I20260927 15:11:37.032506 18641 catalog_manager.cc:5495] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47ec8925c2694970a89ef96982266211 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "47ec8925c2694970a89ef96982266211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47ec8925c2694970a89ef96982266211" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33381 } health_report { overall_health: HEALTHY } } }
I20260927 15:11:37.053848 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.048090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.0.97.2:33381)
0927 15:11:37.048263 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.048268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.048286 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:11:37.048511 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.048515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.048598 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.048744 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:37.049185 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.049997 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.051597 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.052074 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.052188 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.052682 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:37.053238 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:37.053368 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.053569 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.053612 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:11:37.062414 18787 raft_consensus.cc:1261] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 1 FOLLOWER]: Refusing update from remote peer 47ec8925c2694970a89ef96982266211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:37.062476 19055 raft_consensus.cc:1261] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 1 FOLLOWER]: Refusing update from remote peer 47ec8925c2694970a89ef96982266211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:37.063388 19132 consensus_queue.cc:1035] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6485b55cd74fe0be04c06fc08ab187" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:37.063769 19125 consensus_queue.cc:1035] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4fb806371349c5a453cf8af2ab8c1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:37.076242 19135 mvcc.cc:205] Tried to move back new op lower bound from 7333977690357817344 to 7333977690226335744. Current Snapshot: MvccSnapshot[applied={T|T < 7333977690357817344}]
I20260927 15:11:37.090956 18641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:37.091336 18641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:37.095252 18641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c5003549a704bf19a7254127a996fe6: Sending DeleteTablet for 3 replicas of tablet 46a3751df5f846c8859e5b40bf822a96
I20260927 15:11:37.096611 19035 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a3751df5f846c8859e5b40bf822a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36756
I20260927 15:11:37.096586 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a3751df5f846c8859e5b40bf822a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53930
I20260927 15:11:37.097389 18901 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a3751df5f846c8859e5b40bf822a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46658
I20260927 15:11:37.098131 19145 tablet_replica.cc:330] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: stopping tablet replica
I20260927 15:11:37.098577 19145 raft_consensus.cc:2229] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:37.099092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18704
I20260927 15:11:37.099131 19145 raft_consensus.cc:2258] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:37.099822 19146 tablet_replica.cc:330] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: stopping tablet replica
I20260927 15:11:37.100242 19146 raft_consensus.cc:2229] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:37.100699 19146 raft_consensus.cc:2258] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:37.103153 19145 ts_tablet_manager.cc:1905] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:37.107903 19146 ts_tablet_manager.cc:1905] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:37.110703 19145 ts_tablet_manager.cc:1918] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:37.110925 19145 log.cc:1196] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/46a3751df5f846c8859e5b40bf822a96
I20260927 15:11:37.111531 19145 ts_tablet_manager.cc:1939] T 46a3751df5f846c8859e5b40bf822a96 P 2f6485b55cd74fe0be04c06fc08ab187: Deleting consensus metadata
I20260927 15:11:37.113620 18629 catalog_manager.cc:4841] TS 2f6485b55cd74fe0be04c06fc08ab187 (127.0.97.3:46629): tablet 46a3751df5f846c8859e5b40bf822a96 (table test-table [id=f49f56df20594119a6dfedd81a89a95b]) successfully deleted
I20260927 15:11:37.115273 19147 tablet_replica.cc:330] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: stopping tablet replica
I20260927 15:11:37.115779 19147 raft_consensus.cc:2229] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:37.116560 19147 raft_consensus.cc:2258] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:37.118933 19147 ts_tablet_manager.cc:1905] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:37.120463 19146 ts_tablet_manager.cc:1918] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:37.120661 19146 log.cc:1196] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/46a3751df5f846c8859e5b40bf822a96
I20260927 15:11:37.121254 19146 ts_tablet_manager.cc:1939] T 46a3751df5f846c8859e5b40bf822a96 P dc4fb806371349c5a453cf8af2ab8c1d: Deleting consensus metadata
I20260927 15:11:37.123397 18628 catalog_manager.cc:4841] TS dc4fb806371349c5a453cf8af2ab8c1d (127.0.97.1:43281): tablet 46a3751df5f846c8859e5b40bf822a96 (table test-table [id=f49f56df20594119a6dfedd81a89a95b]) successfully deleted
Sep 27 15:11:37 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:11:37.127468 19147 ts_tablet_manager.cc:1918] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:37.127648 19147 log.cc:1196] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/46a3751df5f846c8859e5b40bf822a96
I20260927 15:11:37.128233 19147 ts_tablet_manager.cc:1939] T 46a3751df5f846c8859e5b40bf822a96 P 47ec8925c2694970a89ef96982266211: Deleting consensus metadata
I20260927 15:11:37.130169 18629 catalog_manager.cc:4841] TS 47ec8925c2694970a89ef96982266211 (127.0.97.2:33381): tablet 46a3751df5f846c8859e5b40bf822a96 (table test-table [id=f49f56df20594119a6dfedd81a89a95b]) successfully deleted
I20260927 15:11:37.132870 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.102970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.0.97.1:43281)
0927 15:11:37.103209 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.103218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.113326 (+ 10108us) server_negotiation.cc:407] Connection header received
0927 15:11:37.114052 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.114060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.114178 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.114278 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:37.116417 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.117562 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.119841 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.120507 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.120624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.126111 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:37.126124 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:37.126139 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:37.126184 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:37.129457 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:37.130439 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:37.130457 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:37.130460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:37.130605 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:37.131160 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:37.131166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:37.131170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:37.131670 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:37.132020 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.132250 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.132508 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:11:37.133605 18827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:37.256814 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18704
I20260927 15:11:37.275231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18838
Sep 27 15:11:37 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:11:37.292249 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.276043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.97.2:33381)
0927 15:11:37.276263 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.276268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.276286 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:11:37.276443 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.276447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.276510 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.276592 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:37.278293 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.279057 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.280533 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.280969 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.281115 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.285763 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:37.285793 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:37.285810 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:37.285878 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:37.289198 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:37.289935 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:37.289948 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:37.289950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:37.290107 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:37.290510 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:37.290515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:37.290518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:37.290900 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:37.291246 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.291648 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.291884 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:11:37.293008 18961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:37.410168 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18838
I20260927 15:11:37.429534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18972
Sep 27 15:11:37 dist-test-slave-g0w1 krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:11:37.446379 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.430302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.0.97.3:46629)
0927 15:11:37.430532 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.430537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.430559 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:11:37.430667 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.430672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.430753 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.430872 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:37.432206 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.433022 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.434429 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.434877 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.435026 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.439591 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:37.439609 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:37.439626 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:37.439667 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:37.442864 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:37.444019 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:37.444027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:37.444030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:37.444192 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:37.444625 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:37.444630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:37.444632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:37.445121 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:37.445377 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.445737 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.445929 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:11:37.447178 19095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:37.541563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18972
I20260927 15:11:37.559279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 18611
I20260927 15:11:37.560425 18673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:37.673517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 18611
2026-09-27T15:11:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4814 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:37Z Disabled control of system clock
I20260927 15:11:37.754427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41043
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:38.115384 19156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:38.115697 19156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:38.115783 19156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:38.123602 19156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:38.123728 19156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:38.123766 19156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:38.123826 19156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:38.123876 19156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:38.133880 19156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:38.135285 19156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:38.137158 19156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:38.151362 19161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:38.151355 19162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:38.151484 19164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:38.152530 19156 server_base.cc:1034] running on GCE node
I20260927 15:11:38.153398 19156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:38.154935 19156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:38.156173 19156 hybrid_clock.cc:648] HybridClock initialized: now 1790521898156120 us; error 42 us; skew 500 ppm
I20260927 15:11:38.156525 19156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:38.159057 19156 webserver.cc:460] Webserver started at http://127.0.97.62:34701/ using document root <none> and password file <none>
I20260927 15:11:38.159853 19156 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:38.159955 19156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:38.160229 19156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:38.163370 19156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "eea0d9a794064258baf3e623f7870dea"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-g0w1"
server_key: "bdbdade5cd9e08b65aa56e5743ec4ba1"
server_key_iv: "14d8b22aaa4809c09402530a00f9eb77"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.164204 19156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "eea0d9a794064258baf3e623f7870dea"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-g0w1"
server_key: "bdbdade5cd9e08b65aa56e5743ec4ba1"
server_key_iv: "14d8b22aaa4809c09402530a00f9eb77"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.169775 19156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:11:38.173758 19170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:38.175505 19156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:11:38.175715 19156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "eea0d9a794064258baf3e623f7870dea"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-g0w1"
server_key: "bdbdade5cd9e08b65aa56e5743ec4ba1"
server_key_iv: "14d8b22aaa4809c09402530a00f9eb77"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.175962 19156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:38.189760 19156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:38.190696 19156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:38.191048 19156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:38.210767 19156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41043
I20260927 15:11:38.210775 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41043 every 8 connection(s)
I20260927 15:11:38.212292 19156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:38.216369 19222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:38.221858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19156
I20260927 15:11:38.222036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:38.222546 388 external_mini_cluster.cc:1428] Setting key 979787cfe7b4229c708f447d69c6618b
I20260927 15:11:38.228664 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea: Bootstrap starting.
I20260927 15:11:38.232075 19222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:38.233208 19222 log.cc:824] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:38.236961 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea: No bootstrap required, opened a new log
I20260927 15:11:38.237670 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:38.224620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.97.62:41043)
0927 15:11:38.225719 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:38.225734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:38.225782 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:11:38.227720 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:38.227747 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:38.228277 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:38.228711 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:38.229342 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:38.230750 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:38.232269 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:38.233013 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:38.233236 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:38.234559 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:38.234567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:38.234584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:38.235399 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:38.235505 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:38.235538 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:38.235668 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:38.235950 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 15:11:38.236312 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":346,"threads_started":1}
I20260927 15:11:38.244987 19222 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea0d9a794064258baf3e623f7870dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41043 } }
I20260927 15:11:38.245369 19222 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:38.245450 19222 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea0d9a794064258baf3e623f7870dea, State: Initialized, Role: FOLLOWER
I20260927 15:11:38.246021 19222 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea0d9a794064258baf3e623f7870dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41043 } }
I20260927 15:11:38.246256 19222 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:38.246402 19222 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:38.246567 19222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:38.248914 19222 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea0d9a794064258baf3e623f7870dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41043 } }
I20260927 15:11:38.249447 19222 leader_election.cc:302] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea0d9a794064258baf3e623f7870dea; no voters:
I20260927 15:11:38.250099 19222 leader_election.cc:288] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:38.250272 19227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:38.251122 19227 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [term 1 LEADER]: Becoming Leader. State: Replica: eea0d9a794064258baf3e623f7870dea, State: Running, Role: LEADER
I20260927 15:11:38.251709 19227 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea0d9a794064258baf3e623f7870dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41043 } }
I20260927 15:11:38.252475 19222 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:38.254364 19229 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea0d9a794064258baf3e623f7870dea. Latest consensus state: current_term: 1 leader_uuid: "eea0d9a794064258baf3e623f7870dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea0d9a794064258baf3e623f7870dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41043 } } }
I20260927 15:11:38.254770 19229 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:38.258638 19233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:38.258512 19228 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea0d9a794064258baf3e623f7870dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea0d9a794064258baf3e623f7870dea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41043 } } }
I20260927 15:11:38.258919 19228 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:38.264684 19233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:38.274348 19233 catalog_manager.cc:1347] Generated new cluster ID: 056ca3638e554a29bef6af5d352c6a85
I20260927 15:11:38.274483 19233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:38.287870 19233 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:38.289880 19233 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:38.305188 19233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea: Generated new TSK 0
I20260927 15:11:38.306123 19233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:38.311901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41043
--builtin_ntp_servers=127.0.97.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:38.670181 19247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:38.670498 19247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:38.670590 19247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:38.678807 19247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:38.678926 19247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:38.679037 19247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:38.688987 19247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:38.690292 19247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:38.692215 19247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:38.707240 19252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:38.707232 19255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:38.707224 19253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:38.708712 19247 server_base.cc:1034] running on GCE node
I20260927 15:11:38.709476 19247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:38.710563 19247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:38.711793 19247 hybrid_clock.cc:648] HybridClock initialized: now 1790521898711695 us; error 91 us; skew 500 ppm
I20260927 15:11:38.712168 19247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:38.714919 19247 webserver.cc:460] Webserver started at http://127.0.97.1:42201/ using document root <none> and password file <none>
I20260927 15:11:38.715811 19247 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:38.715924 19247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:38.716311 19247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:38.719810 19247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "51a3691f79ea446ca23bf79007e2447c"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-g0w1"
server_key: "5ae5559122ef3903e108fe0d0a8f4a80"
server_key_iv: "3d6a22944589ccb0ca3de5bd42299413"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.720783 19247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "51a3691f79ea446ca23bf79007e2447c"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-g0w1"
server_key: "5ae5559122ef3903e108fe0d0a8f4a80"
server_key_iv: "3d6a22944589ccb0ca3de5bd42299413"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.726922 19247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 15:11:38.730989 19261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:38.732612 19247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:38.732831 19247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "51a3691f79ea446ca23bf79007e2447c"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-g0w1"
server_key: "5ae5559122ef3903e108fe0d0a8f4a80"
server_key_iv: "3d6a22944589ccb0ca3de5bd42299413"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.733089 19247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:38.764050 19247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:38.764992 19247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:38.765290 19247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:38.766386 19247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:38.768095 19247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:38.768262 19247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:38.768396 19247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:38.768491 19247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:38.804342 19247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39539
I20260927 15:11:38.804388 19373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39539 every 8 connection(s)
I20260927 15:11:38.805723 19247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:38.809790 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19247
I20260927 15:11:38.810019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:38.810472 388 external_mini_cluster.cc:1428] Setting key 70cf7fbb08c51329cb22d42720a560aa
I20260927 15:11:38.813258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41043
--builtin_ntp_servers=127.0.97.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:38.820431 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:38.808602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52169 (local address 127.0.97.62:41043)
0927 15:11:38.809104 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:38.809109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:38.811010 (+ 1901us) server_negotiation.cc:407] Connection header received
0927 15:11:38.812738 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:38.812744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:38.812848 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:38.812970 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:38.814202 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:38.814938 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:38.816657 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:38.817417 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:38.817590 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:38.819336 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:38.819346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:38.819354 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:38.819383 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:38.819439 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:38.819458 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:38.819559 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:38.819820 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 15:11:38.819953 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260927 15:11:38.821452 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:38.809319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41043 (local address 127.0.97.1:52169)
0927 15:11:38.810409 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:38.810459 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:38.812459 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:38.813229 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:38.813239 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:38.813337 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:38.814020 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:38.814039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:38.815142 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:38.815147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:38.816449 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:38.816464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:38.817679 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:38.817687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:38.818029 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:38.818902 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:38.818907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:38.818964 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:38.819105 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:38.819566 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:38.819578 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:38.819589 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:11:38.819762 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 15:11:38.820204 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":350,"threads_started":1}
I20260927 15:11:38.823279 19374 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41043
I20260927 15:11:38.823642 19374 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:38.824474 19374 heartbeater.cc:511] Master 127.0.97.62:41043 requested a full tablet report, sending...
I20260927 15:11:38.827153 19187 ts_manager.cc:194] Registered new tserver with Master: 51a3691f79ea446ca23bf79007e2447c (127.0.97.1:39539)
I20260927 15:11:38.829526 19187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52169
W20260927 15:11:39.179821 19378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:39.180174 19378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:39.180327 19378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:39.188083 19378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:39.188201 19378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:39.188310 19378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:39.198422 19378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:39.199817 19378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:39.201609 19378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:39.216159 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:39.216271 19386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:39.216799 19384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:39.216827 19378 server_base.cc:1034] running on GCE node
I20260927 15:11:39.217597 19378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:39.218729 19378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:39.220010 19378 hybrid_clock.cc:648] HybridClock initialized: now 1790521899219943 us; error 51 us; skew 500 ppm
I20260927 15:11:39.220439 19378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:39.223245 19378 webserver.cc:460] Webserver started at http://127.0.97.2:35997/ using document root <none> and password file <none>
I20260927 15:11:39.224047 19378 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:39.224197 19378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:39.224562 19378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:39.227871 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dce890105a1449f8761189cab2c8621"
format_stamp: "Formatted at 2026-09-27 15:11:39 on dist-test-slave-g0w1"
server_key: "ecd2603dc0e7d48fcdcde09c5c8f3a20"
server_key_iv: "dce03d0af68e2bd88c0f2fb5c4fc79f3"
server_key_version: "encryptionkey@0"
I20260927 15:11:39.228781 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dce890105a1449f8761189cab2c8621"
format_stamp: "Formatted at 2026-09-27 15:11:39 on dist-test-slave-g0w1"
server_key: "ecd2603dc0e7d48fcdcde09c5c8f3a20"
server_key_iv: "dce03d0af68e2bd88c0f2fb5c4fc79f3"
server_key_version: "encryptionkey@0"
I20260927 15:11:39.234776 19378 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:39.238897 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:39.240471 19378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:39.240706 19378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "1dce890105a1449f8761189cab2c8621"
format_stamp: "Formatted at 2026-09-27 15:11:39 on dist-test-slave-g0w1"
server_key: "ecd2603dc0e7d48fcdcde09c5c8f3a20"
server_key_iv: "dce03d0af68e2bd88c0f2fb5c4fc79f3"
server_key_version: "encryptionkey@0"
I20260927 15:11:39.240962 19378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:39.258263 19378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:39.259301 19378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:39.259621 19378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:39.260821 19378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:39.262565 19378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:39.262697 19378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:39.262853 19378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:39.262952 19378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:39.299831 19378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41313
I20260927 15:11:39.299896 19504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41313 every 8 connection(s)
I20260927 15:11:39.301429 19378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:39.302929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19378
I20260927 15:11:39.303160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:39.303643 388 external_mini_cluster.cc:1428] Setting key c6f84a17eacdfea5e7e7cab676a5100a
I20260927 15:11:39.306653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41043
--builtin_ntp_servers=127.0.97.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:39.317093 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.304792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45275 (local address 127.0.97.62:41043)
0927 15:11:39.305047 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:39.305055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:39.306827 (+ 1772us) server_negotiation.cc:407] Connection header received
0927 15:11:39.309033 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:39.309041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:39.309140 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:39.309262 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:39.310533 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.311810 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.313584 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.314170 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.314282 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.316193 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:39.316199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:39.316202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:39.316225 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:39.316259 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:39.316271 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:39.316376 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:39.316630 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 15:11:39.316746 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:11:39.318349 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.305330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41043 (local address 127.0.97.2:45275)
0927 15:11:39.306621 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:39.306672 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:39.308694 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:39.309541 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:39.309552 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:39.309657 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:39.310322 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.310340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.312029 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.312033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.313394 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.313410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.314460 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.314467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.314877 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.315799 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:39.315802 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:39.315859 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:39.315950 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:39.316373 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:39.316386 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:39.316397 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:11:39.316603 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 15:11:39.317093 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":429,"threads_started":1}
I20260927 15:11:39.320001 19505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41043
I20260927 15:11:39.320369 19505 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:39.321384 19505 heartbeater.cc:511] Master 127.0.97.62:41043 requested a full tablet report, sending...
I20260927 15:11:39.323261 19187 ts_manager.cc:194] Registered new tserver with Master: 1dce890105a1449f8761189cab2c8621 (127.0.97.2:41313)
I20260927 15:11:39.324120 19187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45275
W20260927 15:11:39.656872 19508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:39.657181 19508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:39.657344 19508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:39.664798 19508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:39.664907 19508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:39.665081 19508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:39.674324 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:39.675712 19508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:39.677608 19508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:39.692147 19513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:39.692257 19516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:39.692471 19514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:39.692838 19508 server_base.cc:1034] running on GCE node
I20260927 15:11:39.693576 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:39.694589 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:39.695804 19508 hybrid_clock.cc:648] HybridClock initialized: now 1790521899695747 us; error 51 us; skew 500 ppm
I20260927 15:11:39.696189 19508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:39.698738 19508 webserver.cc:460] Webserver started at http://127.0.97.3:35599/ using document root <none> and password file <none>
I20260927 15:11:39.699581 19508 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:39.699718 19508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:39.700057 19508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:39.703229 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "2d075d93b5e24486947da04b342d8fc6"
format_stamp: "Formatted at 2026-09-27 15:11:39 on dist-test-slave-g0w1"
server_key: "566128c8eb7793afe4a9e794015c4202"
server_key_iv: "3e9b38ec2cbfe2c6b99405e3344a2414"
server_key_version: "encryptionkey@0"
I20260927 15:11:39.704100 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2d075d93b5e24486947da04b342d8fc6"
format_stamp: "Formatted at 2026-09-27 15:11:39 on dist-test-slave-g0w1"
server_key: "566128c8eb7793afe4a9e794015c4202"
server_key_iv: "3e9b38ec2cbfe2c6b99405e3344a2414"
server_key_version: "encryptionkey@0"
I20260927 15:11:39.709632 19508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:11:39.713526 19522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:39.715000 19508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:39.715235 19508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "2d075d93b5e24486947da04b342d8fc6"
format_stamp: "Formatted at 2026-09-27 15:11:39 on dist-test-slave-g0w1"
server_key: "566128c8eb7793afe4a9e794015c4202"
server_key_iv: "3e9b38ec2cbfe2c6b99405e3344a2414"
server_key_version: "encryptionkey@0"
I20260927 15:11:39.715523 19508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:39.741968 19508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:39.742901 19508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:39.743270 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:39.744338 19508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:39.746127 19508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:39.746250 19508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:39.746387 19508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:39.746479 19508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:39.781580 19508 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42567
I20260927 15:11:39.781642 19634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42567 every 8 connection(s)
I20260927 15:11:39.783031 19508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:39.786121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19508
I20260927 15:11:39.786284 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:39.786780 388 external_mini_cluster.cc:1428] Setting key 7c4b02e2c15db985ce83cdbe2b766828
I20260927 15:11:39.796818 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.785769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58625 (local address 127.0.97.62:41043)
0927 15:11:39.786009 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:39.786016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:39.787717 (+ 1701us) server_negotiation.cc:407] Connection header received
0927 15:11:39.789954 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:39.789960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:39.790040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:39.790128 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:39.791398 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.792150 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.793916 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.794328 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.794481 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.796017 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:39.796025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:39.796030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:39.796053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:39.796088 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:39.796101 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:39.796186 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:39.796473 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:11:39.796560 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:11:39.797859 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.786402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41043 (local address 127.0.97.3:58625)
0927 15:11:39.787539 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:39.787594 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:39.789688 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:39.790401 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:39.790410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:39.790518 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:39.791234 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.791255 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.792319 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.792325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.793750 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.793764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.794568 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.794574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.794789 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.795602 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:39.795605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:39.795650 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:39.795741 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:39.796199 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:39.796206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:39.796213 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:11:39.796364 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:11:39.796707 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":345,"threads_started":1}
I20260927 15:11:39.799564 19635 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41043
I20260927 15:11:39.799913 19635 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:39.800887 19635 heartbeater.cc:511] Master 127.0.97.62:41043 requested a full tablet report, sending...
I20260927 15:11:39.802718 19187 ts_manager.cc:194] Registered new tserver with Master: 2d075d93b5e24486947da04b342d8fc6 (127.0.97.3:42567)
I20260927 15:11:39.803812 19187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:58625
I20260927 15:11:39.815289 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:39.826138 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.819740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.0.97.62:41043)
0927 15:11:39.819965 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:39.819970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:39.820421 (+ 451us) server_negotiation.cc:407] Connection header received
0927 15:11:39.820561 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:39.820566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:39.820652 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:39.820777 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:39.821250 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.821954 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.823547 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.823974 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.824132 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.825393 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:39.825398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:39.825401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:39.825420 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:39.825463 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:39.825475 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:39.825595 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:39.825827 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 15:11:39.825887 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:11:39.833369 19374 heartbeater.cc:503] Master 127.0.97.62:41043 was elected leader, sending a full tablet report...
I20260927 15:11:39.830745 19187 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:39.838886 19187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:39.863428 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.852128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36654)
0927 15:11:39.852938 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:39.852990 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:39.853178 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:39.854498 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:39.854510 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:39.854554 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:39.855114 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.855133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.857652 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.857660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.860195 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.860215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.862284 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.862292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.862603 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.862660 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:11:39.862799 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 15:11:39.863001 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":52,"thread_start_us":310,"threads_started":1}
I20260927 15:11:39.864320 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.852263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.0.97.1:39539)
0927 15:11:39.852881 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:39.852891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:39.853107 (+ 216us) server_negotiation.cc:407] Connection header received
0927 15:11:39.853474 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:39.853482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:39.853810 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:39.854128 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:39.855365 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.857495 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.860449 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.862073 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.862215 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.862511 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:39.863742 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0927 15:11:39.863955 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":276,"threads_started":1}
I20260927 15:11:39.865351 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.853296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42567 (local address 127.0.0.1:46942)
0927 15:11:39.856058 (+ 2762us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:39.856075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:39.856192 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:39.857051 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:39.857055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:39.857078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:39.857480 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.857495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.859538 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.859543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.861537 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.861549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.863644 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.863654 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.863834 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.863865 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:11:39.864893 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0927 15:11:39.865045 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2652,"thread_start_us":2532,"threads_started":1}
I20260927 15:11:39.865478 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.853240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41313 (local address 127.0.0.1:50478)
0927 15:11:39.854110 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:39.854128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:39.854258 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:39.858138 (+ 3880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:39.858146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:39.858174 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:39.858568 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.858577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.860304 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.860310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.862622 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:39.862633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.864701 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.864707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:39.864871 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.864901 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:11:39.864997 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:11:39.865122 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":103,"thread_start_us":1017,"threads_started":1}
I20260927 15:11:39.865731 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.853481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.97.2:41313)
0927 15:11:39.857619 (+ 4138us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:39.857629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:39.857660 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:11:39.857763 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:39.857768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:39.857947 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:39.858153 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:39.858728 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.860140 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.863043 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.864493 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.864629 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.864837 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:39.865055 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 15:11:39.865226 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3954,"thread_start_us":3833,"threads_started":1}
I20260927 15:11:39.866883 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:39.853969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.0.97.3:42567)
0927 15:11:39.855636 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:39.855648 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:39.856339 (+ 691us) server_negotiation.cc:407] Connection header received
0927 15:11:39.856454 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:39.856461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:39.856710 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:39.856906 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:39.857726 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.859362 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.861731 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:39.862986 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:39.863144 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:39.863402 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:39.864047 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 15:11:39.864228 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":1302,"threads_started":1}
I20260927 15:11:39.868276 19309 tablet_service.cc:1467] Processing CreateTablet for tablet c8319e1e828d454abe758e7030ee7392 (DEFAULT_TABLE table=test-table [id=5227d8e8ebfc4af39b568c1b96affa2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:39.869714 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8319e1e828d454abe758e7030ee7392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:39.869419 19440 tablet_service.cc:1467] Processing CreateTablet for tablet c8319e1e828d454abe758e7030ee7392 (DEFAULT_TABLE table=test-table [id=5227d8e8ebfc4af39b568c1b96affa2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:39.870939 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8319e1e828d454abe758e7030ee7392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:39.870378 19570 tablet_service.cc:1467] Processing CreateTablet for tablet c8319e1e828d454abe758e7030ee7392 (DEFAULT_TABLE table=test-table [id=5227d8e8ebfc4af39b568c1b96affa2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:39.871883 19570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8319e1e828d454abe758e7030ee7392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:39.883309 19653 tablet_bootstrap.cc:492] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Bootstrap starting.
I20260927 15:11:39.887100 19654 tablet_bootstrap.cc:492] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Bootstrap starting.
I20260927 15:11:39.887389 19653 tablet_bootstrap.cc:654] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:39.889288 19653 log.cc:824] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:39.891335 19654 tablet_bootstrap.cc:654] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:39.891932 19655 tablet_bootstrap.cc:492] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Bootstrap starting.
I20260927 15:11:39.892995 19654 log.cc:824] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:39.895792 19654 tablet_bootstrap.cc:492] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: No bootstrap required, opened a new log
I20260927 15:11:39.895884 19653 tablet_bootstrap.cc:492] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: No bootstrap required, opened a new log
I20260927 15:11:39.895921 19655 tablet_bootstrap.cc:654] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:39.896126 19654 ts_tablet_manager.cc:1397] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 15:11:39.896201 19653 ts_tablet_manager.cc:1397] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260927 15:11:39.897235 19655 log.cc:824] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:39.901463 19655 tablet_bootstrap.cc:492] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: No bootstrap required, opened a new log
I20260927 15:11:39.901777 19655 ts_tablet_manager.cc:1397] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:11:39.904624 19653 raft_consensus.cc:348] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:39.905018 19653 raft_consensus.cc:374] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:39.905112 19653 raft_consensus.cc:729] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a3691f79ea446ca23bf79007e2447c, State: Initialized, Role: FOLLOWER
I20260927 15:11:39.905768 19653 consensus_queue.cc:260] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:39.906001 19654 raft_consensus.cc:348] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:39.906488 19654 raft_consensus.cc:374] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:39.906587 19654 raft_consensus.cc:729] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dce890105a1449f8761189cab2c8621, State: Initialized, Role: FOLLOWER
I20260927 15:11:39.907297 19654 consensus_queue.cc:260] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:39.907635 19653 ts_tablet_manager.cc:1428] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260927 15:11:39.908674 19505 heartbeater.cc:503] Master 127.0.97.62:41043 was elected leader, sending a full tablet report...
I20260927 15:11:39.909296 19654 ts_tablet_manager.cc:1428] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260927 15:11:39.912501 19655 raft_consensus.cc:348] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:39.913012 19655 raft_consensus.cc:374] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:39.913126 19655 raft_consensus.cc:729] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d075d93b5e24486947da04b342d8fc6, State: Initialized, Role: FOLLOWER
I20260927 15:11:39.913795 19655 consensus_queue.cc:260] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:39.914990 19635 heartbeater.cc:503] Master 127.0.97.62:41043 was elected leader, sending a full tablet report...
I20260927 15:11:39.915681 19655 ts_tablet_manager.cc:1428] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
W20260927 15:11:40.034952 19636 tablet.cc:2369] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:40.053808 19506 tablet.cc:2369] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:40.058280 19375 tablet.cc:2369] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:40.082862 19661 raft_consensus.cc:482] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:40.083254 19661 raft_consensus.cc:504] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:40.085093 19661 leader_election.cc:288] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a3691f79ea446ca23bf79007e2447c (127.0.97.1:39539), 1dce890105a1449f8761189cab2c8621 (127.0.97.2:41313)
I20260927 15:11:40.090992 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.085435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41313 (local address 127.0.97.3:46935)
0927 15:11:40.085611 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.085634 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:40.085769 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:40.086258 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:40.086263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:40.086289 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:40.086550 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:40.086568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.087675 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.087683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:40.089032 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:40.089044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.090167 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.090173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:40.090359 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.090383 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:11:40.090512 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:11:40.090658 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:11:40.090969 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.085692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46935 (local address 127.0.97.2:41313)
0927 15:11:40.085886 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.085892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.085914 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:11:40.086024 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.086028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.086100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.086232 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:40.086765 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.087543 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.089203 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.090001 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.090141 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.090233 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.090539 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.090669 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:11:40.091845 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.085703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47923 (local address 127.0.97.1:39539)
0927 15:11:40.085886 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.085892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.086512 (+ 620us) server_negotiation.cc:407] Connection header received
0927 15:11:40.086733 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.086738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.086812 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.086923 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:40.087508 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.088193 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.089992 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.090781 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.090938 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.091044 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.091480 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.091592 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:11:40.091879 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.085462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.97.3:47923)
0927 15:11:40.086386 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.086403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:40.086565 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:40.086942 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:40.086946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:40.086969 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:40.087344 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:40.087353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.088363 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.088368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:40.089779 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:40.089792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.090993 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.090999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:40.091254 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.091330 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 15:11:40.091435 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:11:40.091568 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":142,"thread_start_us":422,"threads_started":1}
I20260927 15:11:40.092291 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8319e1e828d454abe758e7030ee7392" candidate_uuid: "2d075d93b5e24486947da04b342d8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dce890105a1449f8761189cab2c8621" is_pre_election: true
I20260927 15:11:40.092934 19460 raft_consensus.cc:2454] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d075d93b5e24486947da04b342d8fc6 in term 0.
I20260927 15:11:40.093080 19329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8319e1e828d454abe758e7030ee7392" candidate_uuid: "2d075d93b5e24486947da04b342d8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a3691f79ea446ca23bf79007e2447c" is_pre_election: true
I20260927 15:11:40.093662 19329 raft_consensus.cc:2454] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d075d93b5e24486947da04b342d8fc6 in term 0.
I20260927 15:11:40.093722 19523 leader_election.cc:302] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dce890105a1449f8761189cab2c8621, 2d075d93b5e24486947da04b342d8fc6; no voters:
I20260927 15:11:40.094246 19661 raft_consensus.cc:2781] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:40.094437 19661 raft_consensus.cc:482] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:40.094559 19661 raft_consensus.cc:3037] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:40.096833 19661 raft_consensus.cc:504] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:40.097483 19661 leader_election.cc:288] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [CANDIDATE]: Term 1 election: Requested vote from peers 51a3691f79ea446ca23bf79007e2447c (127.0.97.1:39539), 1dce890105a1449f8761189cab2c8621 (127.0.97.2:41313)
I20260927 15:11:40.098008 19329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8319e1e828d454abe758e7030ee7392" candidate_uuid: "2d075d93b5e24486947da04b342d8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a3691f79ea446ca23bf79007e2447c"
I20260927 15:11:40.098107 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8319e1e828d454abe758e7030ee7392" candidate_uuid: "2d075d93b5e24486947da04b342d8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dce890105a1449f8761189cab2c8621"
I20260927 15:11:40.098241 19329 raft_consensus.cc:3037] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:40.098374 19460 raft_consensus.cc:3037] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:40.101087 19329 raft_consensus.cc:2454] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d075d93b5e24486947da04b342d8fc6 in term 1.
I20260927 15:11:40.101087 19460 raft_consensus.cc:2454] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d075d93b5e24486947da04b342d8fc6 in term 1.
I20260927 15:11:40.101583 19526 leader_election.cc:302] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d075d93b5e24486947da04b342d8fc6, 51a3691f79ea446ca23bf79007e2447c; no voters:
I20260927 15:11:40.101950 19661 raft_consensus.cc:2781] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:40.102420 19661 raft_consensus.cc:686] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 2d075d93b5e24486947da04b342d8fc6, State: Running, Role: LEADER
I20260927 15:11:40.103034 19661 consensus_queue.cc:237] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } }
I20260927 15:11:40.107821 19185 catalog_manager.cc:5495] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d075d93b5e24486947da04b342d8fc6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2d075d93b5e24486947da04b342d8fc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d075d93b5e24486947da04b342d8fc6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:40.169394 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.164138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.97.3:42567)
0927 15:11:40.164315 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.164320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.164337 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:11:40.164422 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.164426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.164509 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.164638 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:40.164998 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.165733 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.167182 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.167613 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.167710 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.168333 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:40.168847 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:40.168985 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.169157 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.169195 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:11:40.178325 19329 raft_consensus.cc:1261] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 1 FOLLOWER]: Refusing update from remote peer 2d075d93b5e24486947da04b342d8fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:40.178342 19460 raft_consensus.cc:1261] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 1 FOLLOWER]: Refusing update from remote peer 2d075d93b5e24486947da04b342d8fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:40.179423 19663 consensus_queue.cc:1035] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a3691f79ea446ca23bf79007e2447c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:40.179805 19661 consensus_queue.cc:1035] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dce890105a1449f8761189cab2c8621" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:40.207293 19185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:40.207609 19185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:40.211439 19185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eea0d9a794064258baf3e623f7870dea: Sending DeleteTablet for 3 replicas of tablet c8319e1e828d454abe758e7030ee7392
I20260927 15:11:40.212791 19440 tablet_service.cc:1514] Processing DeleteTablet for tablet c8319e1e828d454abe758e7030ee7392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:40 UTC) from {username='slave'} at 127.0.0.1:50478
I20260927 15:11:40.213542 19570 tablet_service.cc:1514] Processing DeleteTablet for tablet c8319e1e828d454abe758e7030ee7392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:40 UTC) from {username='slave'} at 127.0.0.1:46942
I20260927 15:11:40.213707 19676 tablet_replica.cc:330] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: stopping tablet replica
I20260927 15:11:40.214133 19676 raft_consensus.cc:2229] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:40.214617 19676 raft_consensus.cc:2258] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:40.215642 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet c8319e1e828d454abe758e7030ee7392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:40 UTC) from {username='slave'} at 127.0.0.1:36654
I20260927 15:11:40.221972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19247
I20260927 15:11:40.227676 19677 tablet_replica.cc:330] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: stopping tablet replica
I20260927 15:11:40.228303 19677 raft_consensus.cc:2229] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:40.228834 19676 ts_tablet_manager.cc:1905] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:40.229156 19677 raft_consensus.cc:2258] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:40.233836 19678 tablet_replica.cc:330] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: stopping tablet replica
I20260927 15:11:40.236930 19676 ts_tablet_manager.cc:1918] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:40.237130 19676 log.cc:1196] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/c8319e1e828d454abe758e7030ee7392
I20260927 15:11:40.237649 19678 raft_consensus.cc:2229] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:40.237774 19676 ts_tablet_manager.cc:1939] T c8319e1e828d454abe758e7030ee7392 P 1dce890105a1449f8761189cab2c8621: Deleting consensus metadata
I20260927 15:11:40.238104 19677 ts_tablet_manager.cc:1905] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:40.239940 19171 catalog_manager.cc:4841] TS 1dce890105a1449f8761189cab2c8621 (127.0.97.2:41313): tablet c8319e1e828d454abe758e7030ee7392 (table test-table [id=5227d8e8ebfc4af39b568c1b96affa2d]) successfully deleted
I20260927 15:11:40.240027 19678 raft_consensus.cc:2258] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:40.243000 19678 ts_tablet_manager.cc:1905] T c8319e1e828d454abe758e7030ee7392 P 51a3691f79ea446ca23bf79007e2447c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:40.247696 19677 ts_tablet_manager.cc:1918] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:40.247892 19677 log.cc:1196] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/c8319e1e828d454abe758e7030ee7392
I20260927 15:11:40.248512 19677 ts_tablet_manager.cc:1939] T c8319e1e828d454abe758e7030ee7392 P 2d075d93b5e24486947da04b342d8fc6: Deleting consensus metadata
I20260927 15:11:40.250471 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.224981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.0.97.1:39539)
0927 15:11:40.225217 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.225225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.225254 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:11:40.225341 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.225349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.225497 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.225631 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:40.234169 (+ 8538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.235404 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.243347 (+ 7943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.244017 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.244140 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.248518 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:40.248534 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:40.248555 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:40.249401 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:40.249490 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:40.249520 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:40.249661 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.249922 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.250066 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:11:40.250420 19174 catalog_manager.cc:4841] TS 2d075d93b5e24486947da04b342d8fc6 (127.0.97.3:42567): tablet c8319e1e828d454abe758e7030ee7392 (table test-table [id=5227d8e8ebfc4af39b568c1b96affa2d]) successfully deleted
I20260927 15:11:40.251623 19369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:40.359210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19247
I20260927 15:11:40.378880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19378
W20260927 15:11:40.378970 19174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39539 (error 108)
I20260927 15:11:40.381713 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.380774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36662)
0927 15:11:40.381324 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.381477 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":309,"threads_started":1}
W20260927 15:11:40.382220 19174 catalog_manager.cc:4584] TS 51a3691f79ea446ca23bf79007e2447c (127.0.97.1:39539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8319e1e828d454abe758e7030ee7392: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
I20260927 15:11:40.392575 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.380056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.97.2:41313)
0927 15:11:40.380269 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.380278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.380301 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:11:40.380528 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.380534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.380685 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.380804 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:40.382077 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.383033 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.384525 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.384934 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.385081 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.390877 (+ 5796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:40.390891 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:40.390912 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:40.391679 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:40.391742 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:40.391766 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:40.391924 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.392085 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.392215 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:11:40.393443 19500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:40.436093 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.435500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36676)
0927 15:11:40.435770 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.435896 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 15:11:40.437508 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.437019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36692)
0927 15:11:40.437225 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.437318 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:11:40.487673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19378
I20260927 15:11:40.498664 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.497991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36698)
0927 15:11:40.498301 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.498425 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 15:11:40.500207 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.499653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36714)
0927 15:11:40.499882 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.499985 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:11:40.507061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19508
I20260927 15:11:40.515347 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.508199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.97.3:42567)
0927 15:11:40.508413 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.508426 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.508454 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:11:40.508547 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.508551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.508642 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.508768 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:40.509244 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.510143 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.511785 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.512286 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.512437 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.513757 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:40.513768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:40.513784 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:40.514432 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:40.514486 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:40.514518 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:40.514685 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.514825 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.514943 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:11:40.516366 19630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:40.580932 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.580420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36720)
0927 15:11:40.580678 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.580781 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 15:11:40.582186 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.581739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39539 (local address 127.0.0.1:36730)
0927 15:11:40.581963 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:40.582047 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 15:11:40.642098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19508
I20260927 15:11:40.662468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19156
I20260927 15:11:40.663825 19217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:40.776805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19156
2026-09-27T15:11:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3103 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:40 dist-test-slave-g0w1 krb5kdc[19688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:40 dist-test-slave-g0w1 krb5kdc[19688](info): set up 2 sockets
Sep 27 15:11:40 dist-test-slave-g0w1 krb5kdc[19688](info): commencing operation
krb5kdc: starting...
W20260927 15:11:41.923496 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:42 dist-test-slave-g0w1 krb5kdc[19688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:11:42.195398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40639
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:42.554611 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:42.554951 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:42.555142 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:42.562750 19701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:42.562885 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:42.562932 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:42.563062 19701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:42.563160 19701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:42.573640 19701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:42.575100 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:42.576550 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:42.590116 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:42.590430 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:42.590413 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:42.591177 19701 server_base.cc:1034] running on GCE node
I20260927 15:11:42.592131 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:42.593775 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:42.595137 19701 hybrid_clock.cc:648] HybridClock initialized: now 1790521902595006 us; error 106 us; skew 500 ppm
I20260927 15:11:42.595598 19701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:42 dist-test-slave-g0w1 krb5kdc[19688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:42.599232 19701 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:11:42.601014 19701 webserver.cc:460] Webserver started at http://127.0.97.62:41257/ using document root <none> and password file <none>
I20260927 15:11:42.601752 19701 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:42.601859 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:42.602139 19701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:42.605331 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "db7fc362c41941b6a3ca71cb66086e7e"
format_stamp: "Formatted at 2026-09-27 15:11:42 on dist-test-slave-g0w1"
server_key: "e7a2eb99a79172a6813402052b3381c3"
server_key_iv: "ba3e7abdeefde83e369a92db2fe4bcc9"
server_key_version: "encryptionkey@0"
I20260927 15:11:42.606107 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "db7fc362c41941b6a3ca71cb66086e7e"
format_stamp: "Formatted at 2026-09-27 15:11:42 on dist-test-slave-g0w1"
server_key: "e7a2eb99a79172a6813402052b3381c3"
server_key_iv: "ba3e7abdeefde83e369a92db2fe4bcc9"
server_key_version: "encryptionkey@0"
I20260927 15:11:42.612093 19701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:11:42.616293 19716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:42.618045 19701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:42.618289 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "db7fc362c41941b6a3ca71cb66086e7e"
format_stamp: "Formatted at 2026-09-27 15:11:42 on dist-test-slave-g0w1"
server_key: "e7a2eb99a79172a6813402052b3381c3"
server_key_iv: "ba3e7abdeefde83e369a92db2fe4bcc9"
server_key_version: "encryptionkey@0"
I20260927 15:11:42.618537 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:42.638244 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:42.643975 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:42.644326 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:42.665259 19701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40639
I20260927 15:11:42.665292 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40639 every 8 connection(s)
I20260927 15:11:42.666878 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:42.671279 19768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:42.672640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19701
I20260927 15:11:42.672847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:42.673395 388 external_mini_cluster.cc:1428] Setting key cd88c1b38dbb588cab1e282f0119abe9
I20260927 15:11:42.685653 19768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e: Bootstrap starting.
Sep 27 15:11:42 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:42.689550 19768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:42.690963 19768 log.cc:824] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:42.694353 19768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e: No bootstrap required, opened a new log
I20260927 15:11:42.699805 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:42.675518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.0.97.62:40639)
0927 15:11:42.676490 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:42.676507 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:42.676557 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:11:42.678611 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:42.678646 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:42.679295 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:42.679818 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:42.681572 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:42.683212 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:42.684868 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:42.685690 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:42.685932 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:42.690971 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:42.690985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:42.691004 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:42.691137 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:42.694683 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:42.695630 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:42.695638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:42.695646 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:42.695846 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:42.696441 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:42.696446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:42.696448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:42.697113 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:42.697473 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:42.698037 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 15:11:42.698546 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":451,"threads_started":1}
I20260927 15:11:42.702895 19768 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fc362c41941b6a3ca71cb66086e7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40639 } }
I20260927 15:11:42.703469 19768 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:42.703609 19768 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7fc362c41941b6a3ca71cb66086e7e, State: Initialized, Role: FOLLOWER
I20260927 15:11:42.704270 19768 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fc362c41941b6a3ca71cb66086e7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40639 } }
I20260927 15:11:42.704504 19768 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:42.704664 19768 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:42.704846 19768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:42.707293 19768 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fc362c41941b6a3ca71cb66086e7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40639 } }
I20260927 15:11:42.707937 19768 leader_election.cc:302] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7fc362c41941b6a3ca71cb66086e7e; no voters:
I20260927 15:11:42.708590 19768 leader_election.cc:288] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:42.708750 19773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:42.710018 19773 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [term 1 LEADER]: Becoming Leader. State: Replica: db7fc362c41941b6a3ca71cb66086e7e, State: Running, Role: LEADER
I20260927 15:11:42.710641 19773 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fc362c41941b6a3ca71cb66086e7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40639 } }
I20260927 15:11:42.711035 19768 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:42.713840 19774 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7fc362c41941b6a3ca71cb66086e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fc362c41941b6a3ca71cb66086e7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40639 } } }
I20260927 15:11:42.714352 19774 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:42.714505 19775 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7fc362c41941b6a3ca71cb66086e7e. Latest consensus state: current_term: 1 leader_uuid: "db7fc362c41941b6a3ca71cb66086e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7fc362c41941b6a3ca71cb66086e7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40639 } } }
I20260927 15:11:42.714871 19775 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:42.716733 19780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:42.722635 19780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:42.734378 19780 catalog_manager.cc:1347] Generated new cluster ID: 2a5176f30ba74f22b360a310df64e2e9
I20260927 15:11:42.734519 19780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:42.750172 19780 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:42.752063 19780 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:42.764386 19780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e: Generated new TSK 0
I20260927 15:11:42.765343 19780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 15:11:42.852083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40639
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:43.242523 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:43.242889 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:43.243132 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:43.250617 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:43.250809 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:43.251042 19794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:43.261828 19794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:43.263393 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:43.264942 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:43.278858 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:43.278746 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:43.278825 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:43.279630 19794 server_base.cc:1034] running on GCE node
I20260927 15:11:43.280331 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:43.281355 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:43.282615 19794 hybrid_clock.cc:648] HybridClock initialized: now 1790521903282560 us; error 50 us; skew 500 ppm
I20260927 15:11:43.282968 19794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:43 dist-test-slave-g0w1 krb5kdc[19688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:43.286662 19794 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 15:11:43.288448 19794 webserver.cc:460] Webserver started at http://127.0.97.1:35707/ using document root <none> and password file <none>
I20260927 15:11:43.289222 19794 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:43.289377 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:43.289713 19794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:43.293112 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "4e72030c05cb4ef8a31c9da0a7519c6a"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-g0w1"
server_key: "308a55e5ad0a9d433dc28cc739d9f933"
server_key_iv: "f91fd5ad3b0c5e734299a23366538a09"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.294013 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4e72030c05cb4ef8a31c9da0a7519c6a"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-g0w1"
server_key: "308a55e5ad0a9d433dc28cc739d9f933"
server_key_iv: "f91fd5ad3b0c5e734299a23366538a09"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.299901 19794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:11:43.303772 19809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:43.305408 19794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:43.305629 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "4e72030c05cb4ef8a31c9da0a7519c6a"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-g0w1"
server_key: "308a55e5ad0a9d433dc28cc739d9f933"
server_key_iv: "f91fd5ad3b0c5e734299a23366538a09"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.305864 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:43.331224 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:43.336212 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:43.336529 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:43.337574 19794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:43.338999 19794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:43.339128 19794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:11:43.339282 19794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:43.339356 19794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:43.374886 19794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33247
I20260927 15:11:43.374986 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33247 every 8 connection(s)
I20260927 15:11:43.376338 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:43.385195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19794
I20260927 15:11:43.385378 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:43.385883 388 external_mini_cluster.cc:1428] Setting key 1aa07fcf8720b76917e8a6ed13f3d319
Sep 27 15:11:43 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:43.401543 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:43.379365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37451 (local address 127.0.97.62:40639)
0927 15:11:43.379872 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:43.379877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:43.380788 (+ 911us) server_negotiation.cc:407] Connection header received
0927 15:11:43.382747 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:43.382755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:43.382858 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:43.382992 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:43.384732 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.385921 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.388575 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.389290 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.389460 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:43.393625 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:43.393635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:43.393644 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:43.393698 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:43.397384 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:43.398662 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:43.398667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:43.398670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:43.398813 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:43.399496 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:43.399500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:43.399502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:43.399801 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:43.399937 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:43.401061 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0927 15:11:43.401274 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":293,"threads_started":1}
I20260927 15:11:43.403131 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:43.379852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40639 (local address 127.0.97.1:37451)
0927 15:11:43.380605 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:43.380653 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:43.382432 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:43.383291 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:43.383299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:43.383887 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:43.384528 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:43.384548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.386167 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.386172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:43.388402 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:43.388420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.389566 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.389572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:43.389810 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:43.390628 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:43.390654 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:43.393287 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:43.397588 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:43.397601 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:43.397625 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:43.398476 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:43.399012 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:43.399020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:43.399027 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:43.399346 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:43.399947 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:43.399959 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:43.400474 (+ 515us) client_negotiation.cc:769] Sending connection context
0927 15:11:43.400817 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 15:11:43.401467 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":286,"threads_started":1}
I20260927 15:11:43.405121 19922 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40639
I20260927 15:11:43.405527 19922 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:43.406404 19922 heartbeater.cc:511] Master 127.0.97.62:40639 requested a full tablet report, sending...
I20260927 15:11:43.409184 19733 ts_manager.cc:194] Registered new tserver with Master: 4e72030c05cb4ef8a31c9da0a7519c6a (127.0.97.1:33247)
I20260927 15:11:43.411628 19733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37451
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 15:11:43.450452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40639
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:43.808564 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:43.808872 19928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:43.809044 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:43.816797 19928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:43.816921 19928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:43.817157 19928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:43.828734 19928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:43.830222 19928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:43.831815 19928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:43.846211 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:43.846316 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:43.846316 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:43.847441 19928 server_base.cc:1034] running on GCE node
I20260927 15:11:43.848107 19928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:43.849097 19928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:43.850343 19928 hybrid_clock.cc:648] HybridClock initialized: now 1790521903850298 us; error 33 us; skew 500 ppm
I20260927 15:11:43.850740 19928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:43 dist-test-slave-g0w1 krb5kdc[19688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:43.854414 19928 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 15:11:43.856313 19928 webserver.cc:460] Webserver started at http://127.0.97.2:35381/ using document root <none> and password file <none>
I20260927 15:11:43.857157 19928 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:43.857306 19928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:43.857672 19928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:43.860926 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "dfa8fde0fa474be4989efb3f268b64f4"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-g0w1"
server_key: "e1e301e6a5de9650f5bef28ea1bc978a"
server_key_iv: "33efa77853b77dc13caea3e2928a207e"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.861790 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dfa8fde0fa474be4989efb3f268b64f4"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-g0w1"
server_key: "e1e301e6a5de9650f5bef28ea1bc978a"
server_key_iv: "33efa77853b77dc13caea3e2928a207e"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.867651 19928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:11:43.871618 19943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:43.873222 19928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:43.873471 19928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "dfa8fde0fa474be4989efb3f268b64f4"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-g0w1"
server_key: "e1e301e6a5de9650f5bef28ea1bc978a"
server_key_iv: "33efa77853b77dc13caea3e2928a207e"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.873718 19928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:43.890647 19928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:43.895773 19928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:43.896231 19928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:43.897369 19928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:43.898829 19928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:43.898944 19928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:43.899159 19928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:43.899251 19928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:11:43.934938 19928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40761
I20260927 15:11:43.935031 20055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40761 every 8 connection(s)
I20260927 15:11:43.936317 19928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:43.938019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 19928
I20260927 15:11:43.938213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:43.938803 388 external_mini_cluster.cc:1428] Setting key cbc92bcc8ff4bc7adf94d8a48b96bda0
Sep 27 15:11:43 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:43.962584 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:43.940032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40639 (local address 127.0.97.2:42995)
0927 15:11:43.940859 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:43.940912 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:43.942669 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:43.943444 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:43.943453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:43.944013 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:43.944604 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:43.944621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.946192 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.946197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:43.947582 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:43.947595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.948888 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.948897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:43.949291 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:43.950138 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:43.950162 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:43.952543 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:43.956258 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:43.956269 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:43.956297 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:43.957241 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:43.957892 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:43.957899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:43.957905 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:43.958239 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:43.959204 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:43.959217 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:43.959865 (+ 648us) client_negotiation.cc:769] Sending connection context
0927 15:11:43.960276 (+ 411us) client_negotiation.cc:240] Negotiation successful
0927 15:11:43.960917 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":344,"threads_started":1}
I20260927 15:11:43.964030 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:43.939353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42995 (local address 127.0.97.62:40639)
0927 15:11:43.939866 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:43.939873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:43.941009 (+ 1136us) server_negotiation.cc:407] Connection header received
0927 15:11:43.942973 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:43.942981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:43.943096 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:43.943220 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:43.944791 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.946022 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.947736 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:43.948575 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:43.948724 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:43.952834 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:43.952839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:43.952843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:43.952931 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:43.956040 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:43.957474 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:43.957481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:43.957485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:43.957732 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:43.958445 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:43.958450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:43.958454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:43.958969 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:43.959185 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:43.963404 (+ 4219us) server_negotiation.cc:299] Negotiation successful
0927 15:11:43.963716 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260927 15:11:43.965325 20056 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40639
I20260927 15:11:43.965683 20056 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:43.966509 20056 heartbeater.cc:511] Master 127.0.97.62:40639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 15:11:43.968221 19733 ts_manager.cc:194] Registered new tserver with Master: dfa8fde0fa474be4989efb3f268b64f4 (127.0.97.2:40761)
I20260927 15:11:43.969112 19733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 15:11:44.003775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40639
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:44.353943 20062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:44.354281 20062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:44.354466 20062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:44.362234 20062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:44.362373 20062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:44.362581 20062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:44.372928 20062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:44.374650 20062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:44.376567 20062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:44.393092 20068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:44.393092 20070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:44.393163 20067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:44.397166 20062 server_base.cc:1034] running on GCE node
I20260927 15:11:44.397923 20062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:44.399263 20062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:44.400578 20062 hybrid_clock.cc:648] HybridClock initialized: now 1790521904400495 us; error 78 us; skew 500 ppm
I20260927 15:11:44.401028 20062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:44 dist-test-slave-g0w1 krb5kdc[19688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:44.405162 20062 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 15:11:44.407204 20062 webserver.cc:460] Webserver started at http://127.0.97.3:35267/ using document root <none> and password file <none>
I20260927 15:11:44.408070 20062 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:44.408247 20062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:44.408630 20062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:44.412439 20062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "c9272fafa262401faa2b59c9abc50124"
format_stamp: "Formatted at 2026-09-27 15:11:44 on dist-test-slave-g0w1"
server_key: "223d8bdbf643f60e81624d48289b3b11"
server_key_iv: "2ed4077985afafda232beefacf7bcd23"
server_key_version: "encryptionkey@0"
I20260927 15:11:44.413395 20062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c9272fafa262401faa2b59c9abc50124"
format_stamp: "Formatted at 2026-09-27 15:11:44 on dist-test-slave-g0w1"
server_key: "223d8bdbf643f60e81624d48289b3b11"
server_key_iv: "2ed4077985afafda232beefacf7bcd23"
server_key_version: "encryptionkey@0"
I20260927 15:11:44.415275 19922 heartbeater.cc:503] Master 127.0.97.62:40639 was elected leader, sending a full tablet report...
I20260927 15:11:44.420149 20062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 15:11:44.424403 20077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:44.426019 20062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:44.426254 20062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "c9272fafa262401faa2b59c9abc50124"
format_stamp: "Formatted at 2026-09-27 15:11:44 on dist-test-slave-g0w1"
server_key: "223d8bdbf643f60e81624d48289b3b11"
server_key_iv: "2ed4077985afafda232beefacf7bcd23"
server_key_version: "encryptionkey@0"
I20260927 15:11:44.426517 20062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:44.459610 20062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:44.465188 20062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:44.465500 20062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:44.466553 20062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:44.468197 20062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:44.468286 20062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:44.468386 20062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:44.468438 20062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:44.504727 20062 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39813
I20260927 15:11:44.504827 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39813 every 8 connection(s)
I20260927 15:11:44.506106 20062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:44.511544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20062
I20260927 15:11:44.511749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:44.512262 388 external_mini_cluster.cc:1428] Setting key 0817a1f1dc69dc24ab48676202b1113b
Sep 27 15:11:44 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:11:44.528882 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.509390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54037 (local address 127.0.97.62:40639)
0927 15:11:44.509919 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.509924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.510989 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 15:11:44.513187 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.513193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.513320 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.513453 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:44.515173 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.515953 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.517862 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.518297 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.518450 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.522325 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:44.522331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:44.522334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:44.522373 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:44.525191 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.526199 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.526203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.526205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.526331 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.526913 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.526923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.526925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.527333 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:44.527500 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.528247 (+ 747us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.528503 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":267,"threads_started":1}
I20260927 15:11:44.530108 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.509928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40639 (local address 127.0.97.3:54037)
0927 15:11:44.510751 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:44.510791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:44.512878 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:44.513736 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:44.513745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:44.514334 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:44.514964 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.514983 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.516150 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.516154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.517697 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.517711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.518562 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.518569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.518827 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.519576 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:44.519600 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:44.522055 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:44.525377 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:44.525388 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:44.525409 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:44.526030 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:44.526460 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:44.526467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:44.526470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:44.526786 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:44.527471 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:44.527480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:44.527865 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 15:11:44.528145 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 15:11:44.528736 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":302,"threads_started":1}
I20260927 15:11:44.531800 20190 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40639
I20260927 15:11:44.532186 20190 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:44.532964 20190 heartbeater.cc:511] Master 127.0.97.62:40639 requested a full tablet report, sending...
I20260927 15:11:44.534523 19733 ts_manager.cc:194] Registered new tserver with Master: c9272fafa262401faa2b59c9abc50124 (127.0.97.3:39813)
I20260927 15:11:44.535434 19733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54037
I20260927 15:11:44.540602 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:44.558954 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.545073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.0.97.62:40639)
0927 15:11:44.545322 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.545333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.545768 (+ 435us) server_negotiation.cc:407] Connection header received
0927 15:11:44.545949 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.545954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.546045 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.546140 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:44.547772 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.548560 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.549971 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.550378 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.550488 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.553020 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:44.553025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:44.553029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:44.553068 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:44.556053 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.556785 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.556789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.556792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.556965 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.557360 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.557363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.557365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.557704 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:44.557891 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.558364 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.558567 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:11:44.563486 19733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:44.570508 19733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:44.609139 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.583916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39813 (local address 127.0.0.1:46898)
0927 15:11:44.584664 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:44.584718 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:44.584899 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:44.600721 (+ 15822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:44.600729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:44.600764 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:44.601227 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.601245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.603459 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.603464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.605646 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.605659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.607816 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.607823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.608054 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.608128 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 15:11:44.608354 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 15:11:44.608583 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":98,"thread_start_us":949,"threads_started":1}
I20260927 15:11:44.609184 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.585136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.0.97.1:33247)
0927 15:11:44.600548 (+ 15412us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.600554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.600580 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:11:44.600718 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.600727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.600992 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.601243 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:44.602165 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.603708 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.606293 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.607732 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.607857 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.608127 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.608645 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.608816 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15249,"thread_start_us":15067,"threads_started":1}
I20260927 15:11:44.609163 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.584507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33247 (local address 127.0.0.1:44842)
0927 15:11:44.599565 (+ 15058us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:44.599588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:44.599731 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:44.601514 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:44.601518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:44.601555 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:44.601974 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.601987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.603851 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.603859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.606110 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.606125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.608248 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.608253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.608425 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.608455 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:11:44.608547 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:11:44.608655 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14891,"mutex_wait_us":55,"thread_start_us":14774,"threads_started":1}
I20260927 15:11:44.610052 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.585863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40761 (local address 127.0.0.1:43362)
0927 15:11:44.601432 (+ 15569us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:44.601451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:44.601667 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:44.602516 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:44.602519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:44.602539 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:44.602815 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.602822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.604808 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.604815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.606791 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.606802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.609057 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.609066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.609267 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.609408 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 15:11:44.609505 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:11:44.609651 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15439,"mutex_wait_us":1,"thread_start_us":16143,"threads_started":1}
I20260927 15:11:44.610409 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.584190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.0.97.3:39813)
0927 15:11:44.599551 (+ 15361us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.599562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.599607 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:11:44.599731 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.599741 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.600121 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.600681 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:44.601511 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.603310 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.605842 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.607580 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.609495 (+ 1915us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.609741 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.609828 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.610027 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15137,"thread_start_us":16144,"threads_started":1}
I20260927 15:11:44.610709 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.586886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.97.2:40761)
0927 15:11:44.599974 (+ 13088us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.599988 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.601558 (+ 1570us) server_negotiation.cc:407] Connection header received
0927 15:11:44.601882 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.601890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.602137 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.602306 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:44.602994 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.604631 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.606959 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.608459 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.608629 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.608818 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.609998 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.610189 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12920,"thread_start_us":12794,"threads_started":1}
I20260927 15:11:44.614388 19857 tablet_service.cc:1467] Processing CreateTablet for tablet 3855a6c531e94b4695b3bfe7582dbc77 (DEFAULT_TABLE table=test-table [id=a32ecabec9f34d6c9740124b85eac904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:44.614382 19991 tablet_service.cc:1467] Processing CreateTablet for tablet 3855a6c531e94b4695b3bfe7582dbc77 (DEFAULT_TABLE table=test-table [id=a32ecabec9f34d6c9740124b85eac904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:44.614272 20125 tablet_service.cc:1467] Processing CreateTablet for tablet 3855a6c531e94b4695b3bfe7582dbc77 (DEFAULT_TABLE table=test-table [id=a32ecabec9f34d6c9740124b85eac904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:44.616055 19991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3855a6c531e94b4695b3bfe7582dbc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:44.616050 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3855a6c531e94b4695b3bfe7582dbc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:44.616073 20125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3855a6c531e94b4695b3bfe7582dbc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:44.629215 20209 tablet_bootstrap.cc:492] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Bootstrap starting.
I20260927 15:11:44.631858 20210 tablet_bootstrap.cc:492] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Bootstrap starting.
I20260927 15:11:44.633651 20209 tablet_bootstrap.cc:654] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:44.634184 20211 tablet_bootstrap.cc:492] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Bootstrap starting.
I20260927 15:11:44.638177 20209 log.cc:824] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:44.639719 20210 tablet_bootstrap.cc:654] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:44.641135 20210 log.cc:824] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:44.641310 20211 tablet_bootstrap.cc:654] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:44.643169 20211 log.cc:824] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:44.645282 20210 tablet_bootstrap.cc:492] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: No bootstrap required, opened a new log
I20260927 15:11:44.645356 20209 tablet_bootstrap.cc:492] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: No bootstrap required, opened a new log
I20260927 15:11:44.645625 20209 ts_tablet_manager.cc:1397] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260927 15:11:44.645619 20210 ts_tablet_manager.cc:1397] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260927 15:11:44.647486 20211 tablet_bootstrap.cc:492] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: No bootstrap required, opened a new log
I20260927 15:11:44.647785 20211 ts_tablet_manager.cc:1397] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260927 15:11:44.653822 20209 raft_consensus.cc:348] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.654271 20209 raft_consensus.cc:374] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:44.654374 20209 raft_consensus.cc:729] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9272fafa262401faa2b59c9abc50124, State: Initialized, Role: FOLLOWER
I20260927 15:11:44.655133 20209 consensus_queue.cc:260] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.656454 20190 heartbeater.cc:503] Master 127.0.97.62:40639 was elected leader, sending a full tablet report...
I20260927 15:11:44.657028 20209 ts_tablet_manager.cc:1428] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 15:11:44.657737 20210 raft_consensus.cc:348] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.658246 20210 raft_consensus.cc:374] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:44.658358 20210 raft_consensus.cc:729] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e72030c05cb4ef8a31c9da0a7519c6a, State: Initialized, Role: FOLLOWER
I20260927 15:11:44.658738 20211 raft_consensus.cc:348] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.659229 20211 raft_consensus.cc:374] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:44.659034 20210 consensus_queue.cc:260] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.659341 20211 raft_consensus.cc:729] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa8fde0fa474be4989efb3f268b64f4, State: Initialized, Role: FOLLOWER
I20260927 15:11:44.660094 20211 consensus_queue.cc:260] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.661002 20210 ts_tablet_manager.cc:1428] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 15:11:44.662070 20056 heartbeater.cc:503] Master 127.0.97.62:40639 was elected leader, sending a full tablet report...
I20260927 15:11:44.663591 20211 ts_tablet_manager.cc:1428] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260927 15:11:44.688591 20057 tablet.cc:2369] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:44.758069 20191 tablet.cc:2369] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:44.879366 19923 tablet.cc:2369] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:44.985498 20215 raft_consensus.cc:482] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:44.985858 20215 raft_consensus.cc:504] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:44.987881 20215 leader_election.cc:288] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa8fde0fa474be4989efb3f268b64f4 (127.0.97.2:40761), 4e72030c05cb4ef8a31c9da0a7519c6a (127.0.97.1:33247)
I20260927 15:11:44.994570 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.988375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33803 (local address 127.0.97.2:40761)
0927 15:11:44.988639 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.988648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.988680 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:11:44.988903 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.988909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.988991 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.989086 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:44.989677 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.990646 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.992488 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.993578 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.993687 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.993766 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.994148 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.994251 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:11:44.994601 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.988075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40761 (local address 127.0.97.3:33803)
0927 15:11:44.988538 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:44.988561 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:44.988698 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:44.989259 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:44.989263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:44.989299 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:44.989519 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.989533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.990785 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.990791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.992316 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.992329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.993757 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.993766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.993959 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.994001 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:11:44.994111 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:11:44.994253 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336}
I20260927 15:11:44.995478 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.988795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37981 (local address 127.0.97.1:33247)
0927 15:11:44.989015 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.989022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.989277 (+ 255us) server_negotiation.cc:407] Connection header received
0927 15:11:44.989369 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.989374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.989448 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.989550 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:44.990225 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.991183 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.993175 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.994279 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.994419 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.994522 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.995020 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.995199 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:11:44.995484 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.988294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33247 (local address 127.0.97.3:37981)
0927 15:11:44.989151 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:44.989165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:44.989273 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:44.989577 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:44.989580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:44.989629 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:44.990049 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.990062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.991372 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.991377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.992993 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:44.993008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.994511 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.994517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:44.994713 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.994749 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:11:44.994966 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:11:44.995151 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":539,"threads_started":1}
I20260927 15:11:44.995811 20011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3855a6c531e94b4695b3bfe7582dbc77" candidate_uuid: "c9272fafa262401faa2b59c9abc50124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa8fde0fa474be4989efb3f268b64f4" is_pre_election: true
I20260927 15:11:44.996385 20011 raft_consensus.cc:2454] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9272fafa262401faa2b59c9abc50124 in term 0.
I20260927 15:11:44.996534 19877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3855a6c531e94b4695b3bfe7582dbc77" candidate_uuid: "c9272fafa262401faa2b59c9abc50124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" is_pre_election: true
I20260927 15:11:44.997143 19877 raft_consensus.cc:2454] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9272fafa262401faa2b59c9abc50124 in term 0.
I20260927 15:11:44.997153 20080 leader_election.cc:302] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9272fafa262401faa2b59c9abc50124, dfa8fde0fa474be4989efb3f268b64f4; no voters:
I20260927 15:11:44.997560 20215 raft_consensus.cc:2781] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:44.997712 20215 raft_consensus.cc:482] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:44.997826 20215 raft_consensus.cc:3037] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:45.000160 20215 raft_consensus.cc:504] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:45.000824 20215 leader_election.cc:288] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [CANDIDATE]: Term 1 election: Requested vote from peers dfa8fde0fa474be4989efb3f268b64f4 (127.0.97.2:40761), 4e72030c05cb4ef8a31c9da0a7519c6a (127.0.97.1:33247)
I20260927 15:11:45.001302 20011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3855a6c531e94b4695b3bfe7582dbc77" candidate_uuid: "c9272fafa262401faa2b59c9abc50124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa8fde0fa474be4989efb3f268b64f4"
I20260927 15:11:45.001535 20011 raft_consensus.cc:3037] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:45.001489 19877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3855a6c531e94b4695b3bfe7582dbc77" candidate_uuid: "c9272fafa262401faa2b59c9abc50124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a"
I20260927 15:11:45.001734 19877 raft_consensus.cc:3037] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:45.003978 20011 raft_consensus.cc:2454] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9272fafa262401faa2b59c9abc50124 in term 1.
I20260927 15:11:45.004015 19877 raft_consensus.cc:2454] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9272fafa262401faa2b59c9abc50124 in term 1.
I20260927 15:11:45.004444 20080 leader_election.cc:302] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9272fafa262401faa2b59c9abc50124, dfa8fde0fa474be4989efb3f268b64f4; no voters:
I20260927 15:11:45.004794 20215 raft_consensus.cc:2781] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:45.005250 20215 raft_consensus.cc:686] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 1 LEADER]: Becoming Leader. State: Replica: c9272fafa262401faa2b59c9abc50124, State: Running, Role: LEADER
I20260927 15:11:45.005753 20215 consensus_queue.cc:237] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } }
I20260927 15:11:45.011121 19732 catalog_manager.cc:5495] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9272fafa262401faa2b59c9abc50124 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c9272fafa262401faa2b59c9abc50124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9272fafa262401faa2b59c9abc50124" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:45.054198 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:45.047687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.97.3:39813)
0927 15:11:45.047936 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:45.047942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:45.047960 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:11:45.048129 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:45.048133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:45.048209 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:45.048367 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:45.048780 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.049574 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.051103 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.051735 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.051894 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:45.052565 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:45.053344 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:45.053506 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:45.053920 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 15:11:45.053992 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:11:45.062911 19877 raft_consensus.cc:1261] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 1 FOLLOWER]: Refusing update from remote peer c9272fafa262401faa2b59c9abc50124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:45.062911 20011 raft_consensus.cc:1261] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 1 FOLLOWER]: Refusing update from remote peer c9272fafa262401faa2b59c9abc50124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:45.064010 20215 consensus_queue.cc:1035] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa8fde0fa474be4989efb3f268b64f4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:45.064328 20219 consensus_queue.cc:1035] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e72030c05cb4ef8a31c9da0a7519c6a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:45.091400 19733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:45.091728 19733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:45.095829 19733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db7fc362c41941b6a3ca71cb66086e7e: Sending DeleteTablet for 3 replicas of tablet 3855a6c531e94b4695b3bfe7582dbc77
I20260927 15:11:45.097136 20125 tablet_service.cc:1514] Processing DeleteTablet for tablet 3855a6c531e94b4695b3bfe7582dbc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46898
I20260927 15:11:45.097174 19857 tablet_service.cc:1514] Processing DeleteTablet for tablet 3855a6c531e94b4695b3bfe7582dbc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44842
I20260927 15:11:45.097229 19991 tablet_service.cc:1514] Processing DeleteTablet for tablet 3855a6c531e94b4695b3bfe7582dbc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43362
I20260927 15:11:45.099304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19794
I20260927 15:11:45.128918 20233 tablet_replica.cc:330] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: stopping tablet replica
I20260927 15:11:45.129593 20233 raft_consensus.cc:2229] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:45.135700 20232 tablet_replica.cc:330] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: stopping tablet replica
I20260927 15:11:45.136404 20232 raft_consensus.cc:2229] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:45.137302 20232 raft_consensus.cc:2258] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:45.137845 20233 raft_consensus.cc:2258] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:45.138878 20231 tablet_replica.cc:330] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: stopping tablet replica
I20260927 15:11:45.139477 20231 raft_consensus.cc:2229] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:45.140112 20231 raft_consensus.cc:2258] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:45.141463 20233 ts_tablet_manager.cc:1905] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:45.142707 20232 ts_tablet_manager.cc:1905] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:11:45 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 15:11:45.149566 20233 ts_tablet_manager.cc:1918] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:45.149761 20233 log.cc:1196] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/3855a6c531e94b4695b3bfe7582dbc77
I20260927 15:11:45.150327 20233 ts_tablet_manager.cc:1939] T 3855a6c531e94b4695b3bfe7582dbc77 P dfa8fde0fa474be4989efb3f268b64f4: Deleting consensus metadata
I20260927 15:11:45.150758 20231 ts_tablet_manager.cc:1905] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:45.151834 20232 ts_tablet_manager.cc:1918] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:45.152132 20232 log.cc:1196] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/3855a6c531e94b4695b3bfe7582dbc77
I20260927 15:11:45.152838 20232 ts_tablet_manager.cc:1939] T 3855a6c531e94b4695b3bfe7582dbc77 P c9272fafa262401faa2b59c9abc50124: Deleting consensus metadata
I20260927 15:11:45.152822 19719 catalog_manager.cc:4841] TS dfa8fde0fa474be4989efb3f268b64f4 (127.0.97.2:40761): tablet 3855a6c531e94b4695b3bfe7582dbc77 (table test-table [id=a32ecabec9f34d6c9740124b85eac904]) successfully deleted
I20260927 15:11:45.155002 19719 catalog_manager.cc:4841] TS c9272fafa262401faa2b59c9abc50124 (127.0.97.3:39813): tablet 3855a6c531e94b4695b3bfe7582dbc77 (table test-table [id=a32ecabec9f34d6c9740124b85eac904]) successfully deleted
I20260927 15:11:45.157431 20231 ts_tablet_manager.cc:1918] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:45.158174 20231 log.cc:1196] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/3855a6c531e94b4695b3bfe7582dbc77
I20260927 15:11:45.158490 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:45.100981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.97.1:33247)
0927 15:11:45.101168 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:45.101175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:45.124722 (+ 23547us) server_negotiation.cc:407] Connection header received
0927 15:11:45.125384 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:45.125395 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:45.125552 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:45.125666 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:45.130497 (+ 4831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.132288 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.137729 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.138672 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.138814 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:45.144461 (+ 5647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:45.144481 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:45.144500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:45.144551 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:45.148503 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:45.154945 (+ 6442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:45.154958 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:45.154962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:45.155263 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:45.155947 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:45.155956 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:45.155961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:45.156589 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:45.156982 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:45.157415 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 15:11:45.157738 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:11:45.158818 20231 ts_tablet_manager.cc:1939] T 3855a6c531e94b4695b3bfe7582dbc77 P 4e72030c05cb4ef8a31c9da0a7519c6a: Deleting consensus metadata
I20260927 15:11:45.159390 19917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:45.254380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19794
I20260927 15:11:45.254599 19720 catalog_manager.cc:4841] TS 4e72030c05cb4ef8a31c9da0a7519c6a (127.0.97.1:33247): tablet 3855a6c531e94b4695b3bfe7582dbc77 (table test-table [id=a32ecabec9f34d6c9740124b85eac904]) successfully deleted
I20260927 15:11:45.278570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19928
Sep 27 15:11:45 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 15:11:45.295507 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:45.279485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.0.97.2:40761)
0927 15:11:45.279714 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:45.279720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:45.279741 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:11:45.279852 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:45.279857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:45.279933 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:45.280049 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:45.281318 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.282036 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.283497 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.283933 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.284084 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:45.289079 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:45.289095 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:45.289112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:45.289154 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:45.292297 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:45.293144 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:45.293152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:45.293155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:45.293292 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:45.293716 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:45.293721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:45.293726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:45.294170 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:45.294495 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:45.294820 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 15:11:45.295124 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:11:45.296424 20051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:45.389716 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19928
I20260927 15:11:45.415369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20062
Sep 27 15:11:45 dist-test-slave-g0w1 krb5kdc[19688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 15:11:45.442198 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:45.416510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.97.3:39813)
0927 15:11:45.416755 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:45.416762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:45.416794 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:11:45.416855 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:45.416859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:45.416928 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:45.417058 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:45.418877 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.419912 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.421907 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.422453 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.422599 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:45.434644 (+ 12045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:45.434661 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:45.434677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:45.434732 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:45.438576 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:45.439637 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:45.439646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:45.439649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:45.439840 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:45.440330 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:45.440336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:45.440342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:45.440775 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:45.441147 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:45.441531 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 15:11:45.441796 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:11:45.443114 20185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:45.570583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20062
I20260927 15:11:45.590364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 19701
I20260927 15:11:45.591617 19763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:45.705214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 19701
2026-09-27T15:11:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:45Z Disabled control of system clock
I20260927 15:11:45.785785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34439
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:46.167140 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:46.167474 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:46.167559 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:46.175400 20242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:46.175530 20242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:46.175573 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:46.175622 20242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:46.175686 20242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:46.185395 20242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:46.186733 20242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:46.188659 20242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:46.202309 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:46.202214 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:46.202253 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:46.203392 20242 server_base.cc:1034] running on GCE node
I20260927 15:11:46.204308 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:46.205860 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:46.207182 20242 hybrid_clock.cc:648] HybridClock initialized: now 1790521906207090 us; error 73 us; skew 500 ppm
I20260927 15:11:46.207661 20242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:46.210505 20242 webserver.cc:460] Webserver started at http://127.0.97.62:39877/ using document root <none> and password file <none>
I20260927 15:11:46.211477 20242 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:46.211578 20242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:46.211937 20242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:46.215296 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "ef633767e96b4963ae8953d350db4d10"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-g0w1"
server_key: "9efa59d8b5c7b473498a3f7ca9b26b0f"
server_key_iv: "1d369fc831ffa7642dd19617bbe2d2cb"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.216216 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef633767e96b4963ae8953d350db4d10"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-g0w1"
server_key: "9efa59d8b5c7b473498a3f7ca9b26b0f"
server_key_iv: "1d369fc831ffa7642dd19617bbe2d2cb"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.222172 20242 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:46.226369 20256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:46.228222 20242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:11:46.228466 20242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "ef633767e96b4963ae8953d350db4d10"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-g0w1"
server_key: "9efa59d8b5c7b473498a3f7ca9b26b0f"
server_key_iv: "1d369fc831ffa7642dd19617bbe2d2cb"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.228749 20242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:46.259634 20242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:46.260761 20242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:46.261125 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:46.282114 20242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34439
I20260927 15:11:46.282177 20307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34439 every 8 connection(s)
I20260927 15:11:46.283661 20242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:46.284107 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20242
I20260927 15:11:46.284260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:46.284878 388 external_mini_cluster.cc:1428] Setting key b4d073f29fed9e5963a0155683984125
I20260927 15:11:46.289031 20308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:46.301061 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:46.287294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.0.97.62:34439)
0927 15:11:46.288496 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:46.288509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:46.288568 (+ 59us) server_negotiation.cc:407] Connection header received
0927 15:11:46.290479 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:46.290508 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:46.291272 (+ 764us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:46.291807 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:46.292512 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:46.293896 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:46.295568 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:46.296334 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:46.296581 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:46.297942 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:46.297950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:46.297972 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:46.298694 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:46.298798 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:46.298829 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:46.299007 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:46.299310 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:11:46.299701 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":505,"threads_started":1}
I20260927 15:11:46.302408 20308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10: Bootstrap starting.
I20260927 15:11:46.306043 20308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:46.307189 20308 log.cc:824] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:46.310410 20308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10: No bootstrap required, opened a new log
I20260927 15:11:46.318451 20308 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef633767e96b4963ae8953d350db4d10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34439 } }
I20260927 15:11:46.318857 20308 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:46.318936 20308 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef633767e96b4963ae8953d350db4d10, State: Initialized, Role: FOLLOWER
I20260927 15:11:46.319547 20308 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef633767e96b4963ae8953d350db4d10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34439 } }
I20260927 15:11:46.319761 20308 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:46.319862 20308 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:46.320036 20308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:46.322703 20308 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef633767e96b4963ae8953d350db4d10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34439 } }
I20260927 15:11:46.323457 20308 leader_election.cc:302] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef633767e96b4963ae8953d350db4d10; no voters:
I20260927 15:11:46.324107 20308 leader_election.cc:288] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:46.324271 20313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:46.325089 20313 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [term 1 LEADER]: Becoming Leader. State: Replica: ef633767e96b4963ae8953d350db4d10, State: Running, Role: LEADER
I20260927 15:11:46.325673 20313 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef633767e96b4963ae8953d350db4d10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34439 } }
I20260927 15:11:46.326385 20308 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:46.328770 20314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef633767e96b4963ae8953d350db4d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef633767e96b4963ae8953d350db4d10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34439 } } }
I20260927 15:11:46.329332 20314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:46.329195 20315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef633767e96b4963ae8953d350db4d10. Latest consensus state: current_term: 1 leader_uuid: "ef633767e96b4963ae8953d350db4d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef633767e96b4963ae8953d350db4d10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34439 } } }
I20260927 15:11:46.329545 20315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:46.332185 20321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:46.338227 20321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:46.348946 20321 catalog_manager.cc:1347] Generated new cluster ID: 67bd55ed84c24cc882b82b1cd4105c20
I20260927 15:11:46.349097 20321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:46.364440 20321 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:46.366367 20321 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:46.378404 20321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10: Generated new TSK 0
I20260927 15:11:46.379464 20321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:46.393028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34439
--builtin_ntp_servers=127.0.97.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:46.768965 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:46.769276 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:46.769448 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:46.777045 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:46.777158 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:46.777347 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 15:11:46.786984 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:46.788472 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:46.790354 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:46.804863 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:46.804876 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:46.805470 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:46.805749 20332 server_base.cc:1034] running on GCE node
I20260927 15:11:46.806471 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:46.807626 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:46.808861 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790521906808788 us; error 63 us; skew 500 ppm
I20260927 15:11:46.809257 20332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:46.811884 20332 webserver.cc:460] Webserver started at http://127.0.97.1:32925/ using document root <none> and password file <none>
I20260927 15:11:46.812675 20332 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:46.812813 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:46.813171 20332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:46.816453 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data/instance:
uuid: "9410d1e491e141b7b8b5d093571741a4"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-g0w1"
server_key: "3d15142bc9dadc907b9abcf8c9eb3ee6"
server_key_iv: "b30088a831553da06c3d895c28902846"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.817345 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9410d1e491e141b7b8b5d093571741a4"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-g0w1"
server_key: "3d15142bc9dadc907b9abcf8c9eb3ee6"
server_key_iv: "b30088a831553da06c3d895c28902846"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.823038 20332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:11:46.827349 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:46.828969 20332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:46.829150 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal
uuid: "9410d1e491e141b7b8b5d093571741a4"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-g0w1"
server_key: "3d15142bc9dadc907b9abcf8c9eb3ee6"
server_key_iv: "b30088a831553da06c3d895c28902846"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.829422 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:46.853225 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:46.854158 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:46.854498 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:46.855674 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:46.857337 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:46.857437 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:46.857601 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:46.857702 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:46.893306 20332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38719
I20260927 15:11:46.893397 20458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38719 every 8 connection(s)
I20260927 15:11:46.894712 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:46.895462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20332
I20260927 15:11:46.895650 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:46.896219 388 external_mini_cluster.cc:1428] Setting key 173f3e01e3f0f6ba51b096d2e3c114cc
I20260927 15:11:46.899576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34439
--builtin_ntp_servers=127.0.97.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:46.909956 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:46.897961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39023 (local address 127.0.97.62:34439)
0927 15:11:46.898449 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:46.898454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:46.900410 (+ 1956us) server_negotiation.cc:407] Connection header received
0927 15:11:46.902278 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:46.902287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:46.902384 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:46.902524 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:46.903877 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:46.905085 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:46.906813 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:46.907308 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:46.907462 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:46.909022 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:46.909027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:46.909030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:46.909050 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:46.909091 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:46.909103 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:46.909215 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:46.909455 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 15:11:46.909577 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
I20260927 15:11:46.911134 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:46.898692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34439 (local address 127.0.97.1:39023)
0927 15:11:46.900224 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:46.900271 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:46.901957 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:46.902782 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:46.902792 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:46.902900 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:46.903674 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:46.903694 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:46.905274 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:46.905278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:46.906627 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:46.906641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:46.907580 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:46.907588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:46.907848 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:46.908625 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:46.908628 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:46.908684 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:46.908782 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:46.909216 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:46.909230 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:46.909240 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 15:11:46.909444 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 15:11:46.909895 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":401,"threads_started":1}
I20260927 15:11:46.912986 20459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34439
I20260927 15:11:46.913352 20459 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:46.914150 20459 heartbeater.cc:511] Master 127.0.97.62:34439 requested a full tablet report, sending...
I20260927 15:11:46.916693 20273 ts_manager.cc:194] Registered new tserver with Master: 9410d1e491e141b7b8b5d093571741a4 (127.0.97.1:38719)
I20260927 15:11:46.919042 20273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39023
W20260927 15:11:47.273092 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:47.273810 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:47.273916 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:47.282797 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:47.282959 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:47.283143 20463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 15:11:47.299253 20463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:47.301216 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:47.303994 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:47.319154 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:47.319432 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:47.319185 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:47.320616 20463 server_base.cc:1034] running on GCE node
I20260927 15:11:47.321405 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:47.322438 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:47.323683 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790521907323632 us; error 38 us; skew 500 ppm
I20260927 15:11:47.324055 20463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:47.326822 20463 webserver.cc:460] Webserver started at http://127.0.97.2:35179/ using document root <none> and password file <none>
I20260927 15:11:47.327729 20463 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:47.327847 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:47.328181 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:47.331418 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data/instance:
uuid: "42224669575848fca9cf4d85bf517ce1"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-g0w1"
server_key: "5b6e717e35ec7ecda1d15dffc7194c30"
server_key_iv: "e264ba46f24f5a1ab0662a7d82487f35"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.332300 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal/instance:
uuid: "42224669575848fca9cf4d85bf517ce1"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-g0w1"
server_key: "5b6e717e35ec7ecda1d15dffc7194c30"
server_key_iv: "e264ba46f24f5a1ab0662a7d82487f35"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.338212 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:47.342504 20477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.344314 20463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:47.344691 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal
uuid: "42224669575848fca9cf4d85bf517ce1"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-g0w1"
server_key: "5b6e717e35ec7ecda1d15dffc7194c30"
server_key_iv: "e264ba46f24f5a1ab0662a7d82487f35"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.344931 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:47.380929 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:47.381861 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:47.382138 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:47.383255 20463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:47.384954 20463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:47.385057 20463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.385133 20463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:47.385190 20463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.420722 20463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40861
I20260927 15:11:47.420816 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40861 every 8 connection(s)
I20260927 15:11:47.422101 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:47.430534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20463
I20260927 15:11:47.430734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:47.431351 388 external_mini_cluster.cc:1428] Setting key 71445b541fc654e78bfb77d5ed33661a
I20260927 15:11:47.434665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34439
--builtin_ntp_servers=127.0.97.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:47.436743 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.424763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41451 (local address 127.0.97.62:34439)
0927 15:11:47.425286 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:47.425291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:47.426456 (+ 1165us) server_negotiation.cc:407] Connection header received
0927 15:11:47.428451 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:47.428456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:47.428560 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:47.428684 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:47.429821 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.430925 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.433067 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.433713 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.433855 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.435646 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:47.435656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:47.435664 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:47.435693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:47.435750 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:47.435769 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:47.435866 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:47.436165 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 15:11:47.436288 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":306,"threads_started":1}
I20260927 15:11:47.437695 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.425269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34439 (local address 127.0.97.2:41451)
0927 15:11:47.426254 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:47.426298 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:47.428142 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:47.428940 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:47.428949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:47.429051 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:47.429646 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.429661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.431121 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.431125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.432886 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.432902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.433937 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.433943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.434398 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.435269 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:47.435272 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:47.435317 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:47.435419 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:47.435887 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:47.435896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:47.435903 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:11:47.436096 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:11:47.436453 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":310,"threads_started":1}
I20260927 15:11:47.439252 20590 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34439
I20260927 15:11:47.439610 20590 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:47.440594 20590 heartbeater.cc:511] Master 127.0.97.62:34439 requested a full tablet report, sending...
I20260927 15:11:47.442214 20273 ts_manager.cc:194] Registered new tserver with Master: 42224669575848fca9cf4d85bf517ce1 (127.0.97.2:40861)
I20260927 15:11:47.443137 20273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41451
W20260927 15:11:47.780931 20594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:47.781250 20594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:47.781339 20594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:47.788849 20594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:47.788975 20594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:47.789088 20594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 15:11:47.798607 20594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:47.800022 20594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:47.801936 20594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:47.816829 20599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:47.816895 20602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:47.816865 20600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:47.817811 20594 server_base.cc:1034] running on GCE node
I20260927 15:11:47.818490 20594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:47.819624 20594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:47.820868 20594 hybrid_clock.cc:648] HybridClock initialized: now 1790521907820762 us; error 84 us; skew 500 ppm
I20260927 15:11:47.821276 20594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:11:47.823891 20594 webserver.cc:460] Webserver started at http://127.0.97.3:39409/ using document root <none> and password file <none>
I20260927 15:11:47.824642 20594 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:47.824760 20594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:47.825111 20594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:47.828384 20594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data/instance:
uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-g0w1"
server_key: "2b4d82b793364c0393ec6d3fd9924893"
server_key_iv: "e7574897da053d22f707d1481bc3b095"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.829282 20594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-g0w1"
server_key: "2b4d82b793364c0393ec6d3fd9924893"
server_key_iv: "e7574897da053d22f707d1481bc3b095"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.835572 20594 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:11:47.839594 20608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.841368 20594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:47.841596 20594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal
uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-g0w1"
server_key: "2b4d82b793364c0393ec6d3fd9924893"
server_key_iv: "e7574897da053d22f707d1481bc3b095"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.841845 20594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:47.865384 20594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:47.866329 20594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:47.866643 20594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:47.867746 20594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:47.869381 20594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:47.869472 20594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.869618 20594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:47.869719 20594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.906648 20594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42543
I20260927 15:11:47.906744 20720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42543 every 8 connection(s)
I20260927 15:11:47.908126 20594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:47.913357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20594
I20260927 15:11:47.913542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:47.914116 388 external_mini_cluster.cc:1428] Setting key 0167a89db91c6629b9c64715f3b862b9
I20260927 15:11:47.922582 20459 heartbeater.cc:503] Master 127.0.97.62:34439 was elected leader, sending a full tablet report...
I20260927 15:11:47.922978 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.911170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36963 (local address 127.0.97.62:34439)
0927 15:11:47.911396 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:47.911402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:47.912945 (+ 1543us) server_negotiation.cc:407] Connection header received
0927 15:11:47.915636 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:47.915642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:47.915744 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:47.915838 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:47.917234 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.918080 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.919836 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.920246 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.920366 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.922023 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:47.922029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:47.922032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:47.922054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:47.922089 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:47.922102 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:47.922186 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:47.922552 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:11:47.922669 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:11:47.924325 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.911762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34439 (local address 127.0.97.3:36963)
0927 15:11:47.912756 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:47.912800 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:47.915231 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:47.916193 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:47.916209 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:47.916352 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:47.917020 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.917055 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.918225 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.918229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.919674 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.919687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.920511 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.920516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.920741 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.921627 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:47.921630 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:47.921675 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:47.921777 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:47.922218 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:47.922227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:47.922234 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:11:47.922538 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 15:11:47.923024 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":328,"threads_started":1}
I20260927 15:11:47.925711 20721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34439
I20260927 15:11:47.926030 20721 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:47.926855 20721 heartbeater.cc:511] Master 127.0.97.62:34439 requested a full tablet report, sending...
I20260927 15:11:47.928663 20273 ts_manager.cc:194] Registered new tserver with Master: dfc99dd3f9914ac7927fd9cc0b89cbc5 (127.0.97.3:42543)
I20260927 15:11:47.929744 20273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36963
I20260927 15:11:47.929911 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:47.941222 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.935288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.0.97.62:34439)
0927 15:11:47.935471 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:47.935476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:47.935830 (+ 354us) server_negotiation.cc:407] Connection header received
0927 15:11:47.935978 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:47.935983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:47.936074 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:47.936169 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:47.936674 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.937406 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.938770 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.939409 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.939515 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.940582 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:47.940587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:47.940591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:47.940611 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:47.940646 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:47.940658 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:47.940733 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:47.940868 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 15:11:47.940952 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:11:47.945303 20273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:47.952586 20273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:47.993628 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.970168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.0.97.1:38719)
0927 15:11:47.983623 (+ 13455us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:47.983638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:47.983683 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:11:47.983819 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:47.983832 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:47.984196 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:47.984483 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:47.986104 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.987984 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.991263 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.992362 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.992520 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.992851 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:47.993095 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:11:47.993268 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13148,"thread_start_us":15030,"threads_started":1}
I20260927 15:11:47.993674 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.969933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.0.1:54862)
0927 15:11:47.982675 (+ 12742us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:47.982706 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:47.982890 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:47.985417 (+ 2527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:47.985425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:47.985470 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:47.985882 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.985900 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.988154 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.988161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.990986 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.991007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.992583 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.992590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.992816 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.992882 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 15:11:47.993033 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:11:47.993256 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12370,"mutex_wait_us":137,"thread_start_us":12185,"threads_started":1}
I20260927 15:11:47.994418 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.970384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42543 (local address 127.0.0.1:41416)
0927 15:11:47.984743 (+ 14359us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:47.984765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:47.984921 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:47.986201 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:47.986205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:47.986244 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:47.986869 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.986883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.988547 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.988553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.990905 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.990925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.993136 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.993142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.993322 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.993377 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 15:11:47.993470 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:11:47.993616 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14241,"mutex_wait_us":28,"thread_start_us":14075,"threads_started":1}
I20260927 15:11:47.994992 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.969672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40861 (local address 127.0.0.1:35672)
0927 15:11:47.981049 (+ 11377us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:47.981114 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:47.981410 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:47.986242 (+ 4832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:47.986246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:47.986278 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:47.986893 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.986906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.988820 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.988830 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.991300 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:47.991313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.993459 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.993467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:47.993640 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.993672 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:11:47.994177 (+ 505us) client_negotiation.cc:240] Negotiation successful
0927 15:11:47.994301 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10975,"mutex_wait_us":56,"thread_start_us":12006,"threads_started":1}
I20260927 15:11:47.995407 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.970456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.0.97.2:40861)
0927 15:11:47.985330 (+ 14874us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:47.985339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:47.985367 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:11:47.985461 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:47.985469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:47.985764 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:47.985981 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:47.987116 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.988671 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.991503 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.992955 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.993102 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.993325 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:47.994692 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0927 15:11:47.994884 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14715,"thread_start_us":14596,"threads_started":1}
I20260927 15:11:47.995667 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:47.971121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.0.97.3:42543)
0927 15:11:47.984258 (+ 13137us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:47.984270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:47.984882 (+ 612us) server_negotiation.cc:407] Connection header received
0927 15:11:47.985231 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:47.985241 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:47.985527 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:47.985743 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:47.987245 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.988388 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.991136 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:47.992572 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:47.992722 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:47.993002 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:47.993766 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 15:11:47.993959 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12941,"thread_start_us":12773,"threads_started":1}
I20260927 15:11:47.998052 20394 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7c923642a6457aa504b7d3f2ba937e (DEFAULT_TABLE table=test-table [id=0194404da1d24504a460eb03ca862ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:47.998423 20656 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7c923642a6457aa504b7d3f2ba937e (DEFAULT_TABLE table=test-table [id=0194404da1d24504a460eb03ca862ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:47.998960 20525 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7c923642a6457aa504b7d3f2ba937e (DEFAULT_TABLE table=test-table [id=0194404da1d24504a460eb03ca862ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:48.000074 20394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7c923642a6457aa504b7d3f2ba937e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:48.000527 20656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7c923642a6457aa504b7d3f2ba937e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:48.000617 20525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7c923642a6457aa504b7d3f2ba937e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:48.012902 20739 tablet_bootstrap.cc:492] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Bootstrap starting.
I20260927 15:11:48.017215 20739 tablet_bootstrap.cc:654] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:48.018385 20739 log.cc:824] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:48.019603 20740 tablet_bootstrap.cc:492] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Bootstrap starting.
I20260927 15:11:48.020707 20741 tablet_bootstrap.cc:492] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Bootstrap starting.
I20260927 15:11:48.024936 20740 tablet_bootstrap.cc:654] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:48.024932 20741 tablet_bootstrap.cc:654] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:48.025307 20739 tablet_bootstrap.cc:492] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: No bootstrap required, opened a new log
I20260927 15:11:48.025631 20739 ts_tablet_manager.cc:1397] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260927 15:11:48.026262 20740 log.cc:824] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:48.026262 20741 log.cc:824] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:48.029243 20740 tablet_bootstrap.cc:492] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: No bootstrap required, opened a new log
I20260927 15:11:48.029549 20740 ts_tablet_manager.cc:1397] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 15:11:48.029635 20741 tablet_bootstrap.cc:492] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: No bootstrap required, opened a new log
I20260927 15:11:48.029934 20741 ts_tablet_manager.cc:1397] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:11:48.037325 20739 raft_consensus.cc:348] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.037895 20739 raft_consensus.cc:374] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:48.038017 20739 raft_consensus.cc:729] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc99dd3f9914ac7927fd9cc0b89cbc5, State: Initialized, Role: FOLLOWER
I20260927 15:11:48.038251 20740 raft_consensus.cc:348] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.038739 20740 raft_consensus.cc:374] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:48.038851 20740 raft_consensus.cc:729] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9410d1e491e141b7b8b5d093571741a4, State: Initialized, Role: FOLLOWER
I20260927 15:11:48.038748 20739 consensus_queue.cc:260] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.039592 20740 consensus_queue.cc:260] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.040061 20721 heartbeater.cc:503] Master 127.0.97.62:34439 was elected leader, sending a full tablet report...
I20260927 15:11:48.040771 20739 ts_tablet_manager.cc:1428] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 15:11:48.041903 20741 raft_consensus.cc:348] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.042416 20741 raft_consensus.cc:374] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:48.042528 20741 raft_consensus.cc:729] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42224669575848fca9cf4d85bf517ce1, State: Initialized, Role: FOLLOWER
I20260927 15:11:48.043229 20741 consensus_queue.cc:260] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.043519 20740 ts_tablet_manager.cc:1428] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 15:11:48.045552 20741 ts_tablet_manager.cc:1428] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 15:11:48.045679 20590 heartbeater.cc:503] Master 127.0.97.62:34439 was elected leader, sending a full tablet report...
W20260927 15:11:48.147764 20460 tablet.cc:2369] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:48.159961 20722 tablet.cc:2369] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:48.174284 20591 tablet.cc:2369] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:48.227662 20746 raft_consensus.cc:482] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:48.228005 20746 raft_consensus.cc:504] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.229720 20746 leader_election.cc:288] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42224669575848fca9cf4d85bf517ce1 (127.0.97.2:40861), dfc99dd3f9914ac7927fd9cc0b89cbc5 (127.0.97.3:42543)
I20260927 15:11:48.237146 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.230185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43511 (local address 127.0.97.2:40861)
0927 15:11:48.230526 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.230536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.231095 (+ 559us) server_negotiation.cc:407] Connection header received
0927 15:11:48.231360 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.231365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.231458 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.231560 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:48.232208 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.233078 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.235403 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.236197 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.236350 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.236455 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.236738 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.236870 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260927 15:11:48.237191 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.229968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40861 (local address 127.0.97.1:43511)
0927 15:11:48.230949 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:48.230969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:48.231186 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:48.231596 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:48.231601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:48.231627 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:48.232064 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:48.232076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.233220 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.233225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:48.235196 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:48.235210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.236397 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.236403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:48.236574 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.236598 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:11:48.236710 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:11:48.236851 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":565,"threads_started":1}
I20260927 15:11:48.237458 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.230259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53855 (local address 127.0.97.3:42543)
0927 15:11:48.230740 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.230748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.231694 (+ 946us) server_negotiation.cc:407] Connection header received
0927 15:11:48.231957 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.231962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.232033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.232135 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:48.232726 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.233670 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.235751 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.236550 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.236689 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.236796 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.237088 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.237218 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260927 15:11:48.237540 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.230189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42543 (local address 127.0.97.1:53855)
0927 15:11:48.231594 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:48.231612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:48.231767 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:48.232148 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:48.232152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:48.232176 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:48.232547 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:48.232556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.233829 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.233836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:48.235562 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:48.235575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.236751 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.236758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:48.236926 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.236955 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:11:48.237057 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:11:48.237193 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":1,"thread_start_us":804,"threads_started":1}
I20260927 15:11:48.238684 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7c923642a6457aa504b7d3f2ba937e" candidate_uuid: "9410d1e491e141b7b8b5d093571741a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" is_pre_election: true
I20260927 15:11:48.238693 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7c923642a6457aa504b7d3f2ba937e" candidate_uuid: "9410d1e491e141b7b8b5d093571741a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42224669575848fca9cf4d85bf517ce1" is_pre_election: true
I20260927 15:11:48.239281 20676 raft_consensus.cc:2454] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9410d1e491e141b7b8b5d093571741a4 in term 0.
I20260927 15:11:48.239288 20545 raft_consensus.cc:2454] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9410d1e491e141b7b8b5d093571741a4 in term 0.
I20260927 15:11:48.240005 20348 leader_election.cc:302] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9410d1e491e141b7b8b5d093571741a4, dfc99dd3f9914ac7927fd9cc0b89cbc5; no voters:
I20260927 15:11:48.240445 20746 raft_consensus.cc:2781] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:48.240603 20746 raft_consensus.cc:482] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:48.240731 20746 raft_consensus.cc:3037] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:48.242908 20746 raft_consensus.cc:504] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.243631 20746 leader_election.cc:288] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [CANDIDATE]: Term 1 election: Requested vote from peers 42224669575848fca9cf4d85bf517ce1 (127.0.97.2:40861), dfc99dd3f9914ac7927fd9cc0b89cbc5 (127.0.97.3:42543)
I20260927 15:11:48.244156 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7c923642a6457aa504b7d3f2ba937e" candidate_uuid: "9410d1e491e141b7b8b5d093571741a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42224669575848fca9cf4d85bf517ce1"
I20260927 15:11:48.244174 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7c923642a6457aa504b7d3f2ba937e" candidate_uuid: "9410d1e491e141b7b8b5d093571741a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5"
I20260927 15:11:48.244383 20545 raft_consensus.cc:3037] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:48.244392 20676 raft_consensus.cc:3037] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:48.246925 20676 raft_consensus.cc:2454] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9410d1e491e141b7b8b5d093571741a4 in term 1.
I20260927 15:11:48.246971 20545 raft_consensus.cc:2454] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9410d1e491e141b7b8b5d093571741a4 in term 1.
I20260927 15:11:48.247434 20348 leader_election.cc:302] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9410d1e491e141b7b8b5d093571741a4, dfc99dd3f9914ac7927fd9cc0b89cbc5; no voters:
I20260927 15:11:48.247804 20746 raft_consensus.cc:2781] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:48.248281 20746 raft_consensus.cc:686] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9410d1e491e141b7b8b5d093571741a4, State: Running, Role: LEADER
I20260927 15:11:48.248874 20746 consensus_queue.cc:237] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } }
I20260927 15:11:48.254204 20272 catalog_manager.cc:5495] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9410d1e491e141b7b8b5d093571741a4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9410d1e491e141b7b8b5d093571741a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9410d1e491e141b7b8b5d093571741a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:48.294688 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.288238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.0.97.1:38719)
0927 15:11:48.288462 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.288467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.288484 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:11:48.288733 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.288737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.288833 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.289021 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:48.289440 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.290167 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.292349 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.292812 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.292925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.293527 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:48.294096 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:48.294220 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.294352 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.294435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:11:48.304523 20545 raft_consensus.cc:1261] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 1 FOLLOWER]: Refusing update from remote peer 9410d1e491e141b7b8b5d093571741a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:48.304522 20676 raft_consensus.cc:1261] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 1 FOLLOWER]: Refusing update from remote peer 9410d1e491e141b7b8b5d093571741a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:48.305752 20750 consensus_queue.cc:1035] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc99dd3f9914ac7927fd9cc0b89cbc5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:48.306139 20746 consensus_queue.cc:1035] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42224669575848fca9cf4d85bf517ce1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:48.319011 20753 mvcc.cc:205] Tried to move back new op lower bound from 7333977736403816448 to 7333977736191954944. Current Snapshot: MvccSnapshot[applied={T|T < 7333977736403816448}]
I20260927 15:11:48.333810 20272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:48.334167 20272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:48.338243 20272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef633767e96b4963ae8953d350db4d10: Sending DeleteTablet for 3 replicas of tablet 4f7c923642a6457aa504b7d3f2ba937e
I20260927 15:11:48.339656 20656 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7c923642a6457aa504b7d3f2ba937e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:48 UTC) from {username='slave'} at 127.0.0.1:41416
I20260927 15:11:48.339659 20525 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7c923642a6457aa504b7d3f2ba937e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:48 UTC) from {username='slave'} at 127.0.0.1:35672
I20260927 15:11:48.340374 20394 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7c923642a6457aa504b7d3f2ba937e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:48 UTC) from {username='slave'} at 127.0.0.1:54862
I20260927 15:11:48.341454 20764 tablet_replica.cc:330] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: stopping tablet replica
I20260927 15:11:48.342135 20764 raft_consensus.cc:2229] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:48.342778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20332
I20260927 15:11:48.343230 20764 raft_consensus.cc:2258] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:48.346382 20764 ts_tablet_manager.cc:1905] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:48.358180 20764 ts_tablet_manager.cc:1918] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:48.358603 20764 log.cc:1196] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-0/wal/wals/4f7c923642a6457aa504b7d3f2ba937e
I20260927 15:11:48.358829 20762 tablet_replica.cc:330] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: stopping tablet replica
I20260927 15:11:48.359366 20764 ts_tablet_manager.cc:1939] T 4f7c923642a6457aa504b7d3f2ba937e P 9410d1e491e141b7b8b5d093571741a4: Deleting consensus metadata
I20260927 15:11:48.359396 20762 raft_consensus.cc:2229] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:48.359848 20763 tablet_replica.cc:330] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: stopping tablet replica
I20260927 15:11:48.359983 20762 raft_consensus.cc:2258] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:48.360392 20763 raft_consensus.cc:2229] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:48.360914 20763 raft_consensus.cc:2258] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:48.362187 20762 ts_tablet_manager.cc:1905] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:48.362543 20257 catalog_manager.cc:4841] TS 9410d1e491e141b7b8b5d093571741a4 (127.0.97.1:38719): tablet 4f7c923642a6457aa504b7d3f2ba937e (table test-table [id=0194404da1d24504a460eb03ca862ebc]) successfully deleted
I20260927 15:11:48.362892 20763 ts_tablet_manager.cc:1905] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:48.365021 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.344868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.0.97.1:38719)
0927 15:11:48.345057 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.345063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.349067 (+ 4004us) server_negotiation.cc:407] Connection header received
0927 15:11:48.349411 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.349420 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.349613 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.349717 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:48.350625 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.352283 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.360686 (+ 8403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.361382 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.361521 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.363211 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:48.363223 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:48.363239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:48.364085 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:48.364150 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:48.364177 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:48.364301 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.364535 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.364659 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:11:48.366263 20454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:48.371126 20763 ts_tablet_manager.cc:1918] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:48.371321 20763 log.cc:1196] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-1/wal/wals/4f7c923642a6457aa504b7d3f2ba937e
I20260927 15:11:48.371778 20763 ts_tablet_manager.cc:1939] T 4f7c923642a6457aa504b7d3f2ba937e P 42224669575848fca9cf4d85bf517ce1: Deleting consensus metadata
I20260927 15:11:48.372326 20762 ts_tablet_manager.cc:1918] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:48.372572 20762 log.cc:1196] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Deleting WAL directory at /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521577514430-388-0/minicluster-data/ts-2/wal/wals/4f7c923642a6457aa504b7d3f2ba937e
I20260927 15:11:48.373176 20762 ts_tablet_manager.cc:1939] T 4f7c923642a6457aa504b7d3f2ba937e P dfc99dd3f9914ac7927fd9cc0b89cbc5: Deleting consensus metadata
I20260927 15:11:48.373476 20260 catalog_manager.cc:4841] TS 42224669575848fca9cf4d85bf517ce1 (127.0.97.2:40861): tablet 4f7c923642a6457aa504b7d3f2ba937e (table test-table [id=0194404da1d24504a460eb03ca862ebc]) successfully deleted
I20260927 15:11:48.375432 20258 catalog_manager.cc:4841] TS dfc99dd3f9914ac7927fd9cc0b89cbc5 (127.0.97.3:42543): tablet 4f7c923642a6457aa504b7d3f2ba937e (table test-table [id=0194404da1d24504a460eb03ca862ebc]) successfully deleted
I20260927 15:11:48.474346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20332
I20260927 15:11:48.493271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20463
I20260927 15:11:48.500906 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.494163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.0.97.2:40861)
0927 15:11:48.494383 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.494389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.494411 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:11:48.494483 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.494489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.494601 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.494696 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:48.495215 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.496114 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.497684 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.498111 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.498235 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.499548 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:48.499558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:48.499573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:48.500205 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:48.500258 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:48.500280 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:48.500376 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.500533 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.500636 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 15:11:48.501658 20585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:48.593688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20463
I20260927 15:11:48.611258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20594
I20260927 15:11:48.617992 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.612129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.97.3:42543)
0927 15:11:48.612292 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.612297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.612315 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:11:48.612407 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.612411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.612498 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.612581 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:48.612936 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.613736 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.615132 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.615571 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.615680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.616719 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:48.616728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:48.616742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:48.617335 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:48.617380 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:48.617401 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:48.617487 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.617640 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.617734 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:48.618724 20716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:48.709295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20594
I20260927 15:11:48.725883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20242
I20260927 15:11:48.727130 20303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:48.839767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20242
2026-09-27T15:11:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3133 ms)
[----------] 6 tests from EncryptionPolicyTest (23805 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:48 dist-test-slave-g0w1 krb5kdc[20772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:48 dist-test-slave-g0w1 krb5kdc[20772](info): set up 2 sockets
Sep 27 15:11:48 dist-test-slave-g0w1 krb5kdc[20772](info): commencing operation
krb5kdc: starting...
W20260927 15:11:50.020110 388 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:50 dist-test-slave-g0w1 krb5kdc[20772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:50.294931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:50.645093 20785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:50.645424 20785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:50.645525 20785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:50.653245 20785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:50.653394 20785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:50.653442 20785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:50.653497 20785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:50.653541 20785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:50.663902 20785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:50.665251 20785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:50.666612 20785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:50.679747 20791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:50.679749 20793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:50.680124 20790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:50.680843 20785 server_base.cc:1034] running on GCE node
I20260927 15:11:50.681723 20785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:50.683357 20785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:50.684610 20785 hybrid_clock.cc:648] HybridClock initialized: now 1790521910684572 us; error 59 us; skew 500 ppm
I20260927 15:11:50.685009 20785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:50 dist-test-slave-g0w1 krb5kdc[20772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:50.692261 20785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:50.694118 20785 webserver.cc:460] Webserver started at http://127.0.0.1:43501/ using document root <none> and password file <none>
I20260927 15:11:50.694936 20785 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:50.695149 20785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:50.695503 20785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:50.698761 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31"
format_stamp: "Formatted at 2026-09-27 15:11:50 on dist-test-slave-g0w1"
server_key: "794df2ecef20b2ec4302287a27a4de94"
server_key_iv: "d5d96fb13d826189d04dc6f3ed054a76"
server_key_version: "encryptionkey@0"
I20260927 15:11:50.699719 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31"
format_stamp: "Formatted at 2026-09-27 15:11:50 on dist-test-slave-g0w1"
server_key: "794df2ecef20b2ec4302287a27a4de94"
server_key_iv: "d5d96fb13d826189d04dc6f3ed054a76"
server_key_version: "encryptionkey@0"
I20260927 15:11:50.705637 20785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:11:50.709910 20800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:50.711760 20785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 15:11:50.711982 20785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31"
format_stamp: "Formatted at 2026-09-27 15:11:50 on dist-test-slave-g0w1"
server_key: "794df2ecef20b2ec4302287a27a4de94"
server_key_iv: "d5d96fb13d826189d04dc6f3ed054a76"
server_key_version: "encryptionkey@0"
I20260927 15:11:50.712215 20785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:50.757033 20785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:50.762070 20785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:50.762409 20785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:50.782960 20785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40283
I20260927 15:11:50.782986 20851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40283 every 8 connection(s)
I20260927 15:11:50.784606 20785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:50.788786 20852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:50.792223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20785
I20260927 15:11:50.792493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:50.793279 388 external_mini_cluster.cc:1428] Setting key 5367d8c6c50a98c6692802500d8ef4be
I20260927 15:11:50.802218 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31: Bootstrap starting.
I20260927 15:11:50.805641 20852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:50.806792 20852 log.cc:824] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:11:50 dist-test-slave-g0w1 krb5kdc[20772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:11:50.810674 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31: No bootstrap required, opened a new log
I20260927 15:11:50.820142 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:50.795605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41551 (local address 127.0.0.1:40283)
0927 15:11:50.796515 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:50.796528 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:50.796576 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:11:50.798454 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:50.798480 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:50.798979 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:50.799552 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:50.801131 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:50.802639 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:50.804118 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:50.804857 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:50.805078 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:50.809939 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:50.809953 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:50.809976 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:50.810028 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:50.814388 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:50.815606 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:50.815622 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:50.815632 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:50.815885 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:50.816432 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:50.816437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:50.816441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:50.817296 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:50.817660 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:50.818109 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:11:50.818531 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":395,"threads_started":1}
I20260927 15:11:50.822376 20852 raft_consensus.cc:348] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260927 15:11:50.822809 20852 raft_consensus.cc:374] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:50.822942 20852 raft_consensus.cc:729] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e512f9d7ae1f4dc09945ab7e60dc5f31, State: Initialized, Role: FOLLOWER
I20260927 15:11:50.823652 20852 consensus_queue.cc:260] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260927 15:11:50.823935 20852 raft_consensus.cc:388] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:50.824098 20852 raft_consensus.cc:482] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:50.824308 20852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:50.825227 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:50.826785 20852 raft_consensus.cc:504] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260927 15:11:50.827427 20852 leader_election.cc:302] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e512f9d7ae1f4dc09945ab7e60dc5f31; no voters:
I20260927 15:11:50.828478 20852 leader_election.cc:288] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:50.828640 20858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:50.830778 20852 sys_catalog.cc:564] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:50.841199 20858 raft_consensus.cc:686] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [term 1 LEADER]: Becoming Leader. State: Replica: e512f9d7ae1f4dc09945ab7e60dc5f31, State: Running, Role: LEADER
I20260927 15:11:50.842128 20858 consensus_queue.cc:237] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260927 15:11:50.845728 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } } }
I20260927 15:11:50.846362 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:50.846129 20862 sys_catalog.cc:455] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e512f9d7ae1f4dc09945ab7e60dc5f31. Latest consensus state: current_term: 1 leader_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e512f9d7ae1f4dc09945ab7e60dc5f31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } } }
I20260927 15:11:50.846496 20862 sys_catalog.cc:458] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:50.857906 20875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:50.860417 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:50.833667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34241 (local address 127.0.0.1:40283)
0927 15:11:50.833937 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:50.833947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:50.840110 (+ 6163us) server_negotiation.cc:407] Connection header received
0927 15:11:50.840352 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:50.840362 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:50.840487 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:50.840647 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:50.843122 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:50.844441 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:50.845897 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:50.846705 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:50.846841 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:50.850708 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:50.850720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:50.850728 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:50.850795 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:50.856025 (+ 5230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:50.857577 (+ 1552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:50.857582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:50.857585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:50.857759 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:50.858472 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:50.858481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:50.858490 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:50.859009 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:50.859365 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:50.859782 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 15:11:50.860000 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:11:50.864138 20875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:50.874889 20875 catalog_manager.cc:1347] Generated new cluster ID: 4b8dbea941104f628b6f2d01043e9c43
I20260927 15:11:50.875034 20875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:50.884493 20875 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:50.886440 20875 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:50.896694 20875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e512f9d7ae1f4dc09945ab7e60dc5f31: Generated new TSK 0
I20260927 15:11:50.897884 20875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:50.906549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20785
I20260927 15:11:50.908031 20847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:51.001210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20785
W20260927 15:11:51.019376 20770 connection.cc:535] client connection to 127.0.0.1:40283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40283): Connection reset by peer (error 104)
2026-09-27T15:11:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:51 dist-test-slave-g0w1 krb5kdc[20884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:51 dist-test-slave-g0w1 krb5kdc[20884](info): set up 2 sockets
Sep 27 15:11:51 dist-test-slave-g0w1 krb5kdc[20884](info): commencing operation
krb5kdc: starting...
W20260927 15:11:52.127564 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:52 dist-test-slave-g0w1 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:52.398134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:52.741644 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:52.742038 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:52.742208 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:52.749727 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:52.749918 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:52.750041 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:52.750149 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:52.750226 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:52.760562 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:52.761952 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:52.763391 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:52.775372 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:52.775323 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:52.775393 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:52.776237 20897 server_base.cc:1034] running on GCE node
I20260927 15:11:52.777089 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:52.778558 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:52.779764 20897 hybrid_clock.cc:648] HybridClock initialized: now 1790521912779694 us; error 58 us; skew 500 ppm
I20260927 15:11:52.780158 20897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:52 dist-test-slave-g0w1 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:52.783622 20897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:52.785269 20897 webserver.cc:460] Webserver started at http://127.0.0.1:45993/ using document root <none> and password file <none>
I20260927 15:11:52.785987 20897 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:52.786144 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:52.786458 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:52.789618 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "56eed7a78a144c268fa5fcdd040e6eca"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-g0w1"
server_key: "29ee4b0ba7af3f21b34824003cb44ffb"
server_key_iv: "02b04ad898671c184ab8c50a71311a60"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.790426 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "56eed7a78a144c268fa5fcdd040e6eca"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-g0w1"
server_key: "29ee4b0ba7af3f21b34824003cb44ffb"
server_key_iv: "02b04ad898671c184ab8c50a71311a60"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.795897 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 15:11:52.799567 20912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:52.800983 20897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:11:52.801180 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "56eed7a78a144c268fa5fcdd040e6eca"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-g0w1"
server_key: "29ee4b0ba7af3f21b34824003cb44ffb"
server_key_iv: "02b04ad898671c184ab8c50a71311a60"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.801393 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:52.813521 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:52.818089 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:52.818398 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:52.837850 20897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42991
I20260927 15:11:52.837898 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42991 every 8 connection(s)
I20260927 15:11:52.839491 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:52.843694 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:52.844257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 20897
I20260927 15:11:52.844448 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:52.844890 388 external_mini_cluster.cc:1428] Setting key 03c461218d85150b99620e2a169e65d1
I20260927 15:11:52.859576 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca: Bootstrap starting.
I20260927 15:11:52.861640 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.846955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59689 (local address 127.0.0.1:42991)
0927 15:11:52.847939 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.847955 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.848010 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:11:52.850722 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.850755 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.851424 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.851931 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.852492 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.854412 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.855989 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.857166 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.857371 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.858653 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.858664 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.858684 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:52.859467 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.859587 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:52.859623 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.859782 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.860070 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.860257 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":360,"threads_started":1}
I20260927 15:11:52.863476 20964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:52.864684 20964 log.cc:824] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:52.865837 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:52.867647 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca: No bootstrap required, opened a new log
I20260927 15:11:52.877179 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.870254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49649 (local address 127.0.0.1:42991)
0927 15:11:52.870447 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.870479 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.870802 (+ 323us) server_negotiation.cc:407] Connection header received
0927 15:11:52.870968 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.870974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.871061 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.871226 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.871715 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.872933 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.874349 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.875090 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.875229 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.876346 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.876352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.876359 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:52.876391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.876465 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:52.876487 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.876628 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.876761 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.876821 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:11:52.878211 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56eed7a78a144c268fa5fcdd040e6eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260927 15:11:52.878641 20964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:52.878772 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eed7a78a144c268fa5fcdd040e6eca, State: Initialized, Role: FOLLOWER
I20260927 15:11:52.879457 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56eed7a78a144c268fa5fcdd040e6eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260927 15:11:52.879721 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:52.879887 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:52.880066 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:52.882273 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56eed7a78a144c268fa5fcdd040e6eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260927 15:11:52.882791 20964 leader_election.cc:302] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56eed7a78a144c268fa5fcdd040e6eca; no voters:
I20260927 15:11:52.883460 20964 leader_election.cc:288] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:52.883605 20974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:52.884912 20974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [term 1 LEADER]: Becoming Leader. State: Replica: 56eed7a78a144c268fa5fcdd040e6eca, State: Running, Role: LEADER
I20260927 15:11:52.885449 20974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56eed7a78a144c268fa5fcdd040e6eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260927 15:11:52.885839 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:52.887957 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56eed7a78a144c268fa5fcdd040e6eca. Latest consensus state: current_term: 1 leader_uuid: "56eed7a78a144c268fa5fcdd040e6eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56eed7a78a144c268fa5fcdd040e6eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } } }
I20260927 15:11:52.888288 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:52.888939 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56eed7a78a144c268fa5fcdd040e6eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56eed7a78a144c268fa5fcdd040e6eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } } }
I20260927 15:11:52.889219 20981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:52.889286 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:52.897934 20981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:52.907867 20981 catalog_manager.cc:1347] Generated new cluster ID: 630c723364c945ca9cdf6e1034f14643
I20260927 15:11:52.907996 20981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:52.925608 20981 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:52.927587 20981 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:52.939630 20981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56eed7a78a144c268fa5fcdd040e6eca: Generated new TSK 0
I20260927 15:11:52.940614 20981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:52.951789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20897
W20260927 15:11:52.952979 20959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59689
W20260927 15:11:52.953914 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 20897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:11:52.954109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 20897
W20260927 15:11:52.971345 20882 connection.cc:535] client connection to 127.0.0.1:42991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42991): Connection reset by peer (error 104)
2026-09-27T15:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:53 dist-test-slave-g0w1 krb5kdc[20995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:53 dist-test-slave-g0w1 krb5kdc[20995](info): set up 2 sockets
Sep 27 15:11:53 dist-test-slave-g0w1 krb5kdc[20995](info): commencing operation
krb5kdc: starting...
W20260927 15:11:54.127728 388 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:54 dist-test-slave-g0w1 krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:54.419296 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:54.770431 21008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:54.770931 21008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:54.771154 21008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:54.779021 21008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:54.779222 21008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:54.779282 21008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:54.779346 21008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:54.779398 21008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:54.790575 21008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:54.792147 21008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:54.793569 21008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:54.807852 21014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:54.808040 21013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:54.807883 21016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:54.809048 21008 server_base.cc:1034] running on GCE node
I20260927 15:11:54.809900 21008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:54.811614 21008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:54.812891 21008 hybrid_clock.cc:648] HybridClock initialized: now 1790521914812841 us; error 49 us; skew 500 ppm
I20260927 15:11:54.813361 21008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:54 dist-test-slave-g0w1 krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:54.816847 21008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:54.818635 21008 webserver.cc:460] Webserver started at http://127.0.0.1:36833/ using document root <none> and password file <none>
I20260927 15:11:54.819605 21008 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:54.819782 21008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:54.820133 21008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:54.823462 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "09b6696d00474bd38c9c7b9baca0a3c4"
format_stamp: "Formatted at 2026-09-27 15:11:54 on dist-test-slave-g0w1"
server_key: "6c10b7cfeecc9d8cb93255a9a37f0a8c"
server_key_iv: "e9c723d9db533cd8776595bfa7bd6d48"
server_key_version: "encryptionkey@0"
I20260927 15:11:54.824291 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "09b6696d00474bd38c9c7b9baca0a3c4"
format_stamp: "Formatted at 2026-09-27 15:11:54 on dist-test-slave-g0w1"
server_key: "6c10b7cfeecc9d8cb93255a9a37f0a8c"
server_key_iv: "e9c723d9db533cd8776595bfa7bd6d48"
server_key_version: "encryptionkey@0"
I20260927 15:11:54.829926 21008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:11:54.834090 21023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:54.835939 21008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:54.836179 21008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "09b6696d00474bd38c9c7b9baca0a3c4"
format_stamp: "Formatted at 2026-09-27 15:11:54 on dist-test-slave-g0w1"
server_key: "6c10b7cfeecc9d8cb93255a9a37f0a8c"
server_key_iv: "e9c723d9db533cd8776595bfa7bd6d48"
server_key_version: "encryptionkey@0"
I20260927 15:11:54.836418 21008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:54.842751 21008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:54.847718 21008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:54.848058 21008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:54.867377 21008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34217
I20260927 15:11:54.867426 21074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34217 every 8 connection(s)
I20260927 15:11:54.868953 21008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:54.873184 21075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:54.875602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21008
I20260927 15:11:54.875782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:54.876386 388 external_mini_cluster.cc:1428] Setting key 463a9de5c4e6b7a693187f83895520a6
I20260927 15:11:54.885028 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:54.878387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35629 (local address 127.0.0.1:34217)
0927 15:11:54.879362 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:54.879378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:54.879427 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:11:54.881289 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:54.881315 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:54.881712 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:54.882160 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:54.883296 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:54.883302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:54.883317 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:54.883333 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:54.883392 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:54.883412 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:54.883665 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:54.883827 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:11:54.884029 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":392,"threads_started":1}
I20260927 15:11:54.886366 21075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4: Bootstrap starting.
I20260927 15:11:54.889322 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:54.890228 21075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:54.891399 21075 log.cc:824] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:54.894906 21075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4: No bootstrap required, opened a new log
I20260927 15:11:54.896579 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:54.894058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59999 (local address 127.0.0.1:34217)
0927 15:11:54.894256 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:54.894264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:54.894592 (+ 328us) server_negotiation.cc:407] Connection header received
0927 15:11:54.894811 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:54.894815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:54.894862 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:54.894974 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:54.895943 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:54.895948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:54.895951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:54.895966 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:54.896003 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:54.896019 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:54.896162 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:54.896257 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 15:11:54.896314 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:11:54.903200 21075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:11:54.903685 21075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:54.903841 21075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b6696d00474bd38c9c7b9baca0a3c4, State: Initialized, Role: FOLLOWER
I20260927 15:11:54.904559 21075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:11:54.904834 21075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:54.904987 21075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:54.905176 21075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:54.907801 21075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:11:54.908325 21075 leader_election.cc:302] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b6696d00474bd38c9c7b9baca0a3c4; no voters:
I20260927 15:11:54.908950 21075 leader_election.cc:288] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:54.909119 21085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:54.909901 21085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [term 1 LEADER]: Becoming Leader. State: Replica: 09b6696d00474bd38c9c7b9baca0a3c4, State: Running, Role: LEADER
I20260927 15:11:54.910468 21085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:11:54.911233 21075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:54.913409 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b6696d00474bd38c9c7b9baca0a3c4. Latest consensus state: current_term: 1 leader_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } } }
I20260927 15:11:54.913800 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:54.915412 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b6696d00474bd38c9c7b9baca0a3c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } } }
I20260927 15:11:54.915782 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:54.915649 21090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:54.921237 21090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:54.930930 21090 catalog_manager.cc:1347] Generated new cluster ID: 8b06b516bbb940238a8b4e220157ea03
I20260927 15:11:54.931124 21090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:54.955058 21090 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:54.955565 21090 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:54.966451 21090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b6696d00474bd38c9c7b9baca0a3c4: Generated new TSK 0
I20260927 15:11:54.967419 21090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:54.980636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21008
W20260927 15:11:54.981892 21070 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35629
W20260927 15:11:54.982682 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:11:54.982832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21008
2026-09-27T15:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:55 dist-test-slave-g0w1 krb5kdc[21106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:55 dist-test-slave-g0w1 krb5kdc[21106](info): set up 2 sockets
Sep 27 15:11:55 dist-test-slave-g0w1 krb5kdc[21106](info): commencing operation
krb5kdc: starting...
W20260927 15:11:56.176069 388 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:56 dist-test-slave-g0w1 krb5kdc[21106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:56.451476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:56.815013 21119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:56.815376 21119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:56.815557 21119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:56.823051 21119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:56.823205 21119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:56.823329 21119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:56.823449 21119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:56.823516 21119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:56.833974 21119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:56.835486 21119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:56.836943 21119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:56.849572 21124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:56.849615 21127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:56.849639 21125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:56.850771 21119 server_base.cc:1034] running on GCE node
I20260927 15:11:56.851758 21119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:56.853370 21119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:56.854600 21119 hybrid_clock.cc:648] HybridClock initialized: now 1790521916854543 us; error 47 us; skew 500 ppm
I20260927 15:11:56.855012 21119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:56 dist-test-slave-g0w1 krb5kdc[21106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:56.858481 21119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:56.860432 21119 webserver.cc:460] Webserver started at http://127.0.0.1:45535/ using document root <none> and password file <none>
I20260927 15:11:56.861289 21119 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:56.861462 21119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:56.861825 21119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:56.865087 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "1aac6f9eee0942588a71dad2c2dac07b"
format_stamp: "Formatted at 2026-09-27 15:11:56 on dist-test-slave-g0w1"
server_key: "7ecb0ffb60d4e1d179cb276ddc2b16c4"
server_key_iv: "2c6026e235ba5002f3de458ac48c31b8"
server_key_version: "encryptionkey@0"
I20260927 15:11:56.865875 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "1aac6f9eee0942588a71dad2c2dac07b"
format_stamp: "Formatted at 2026-09-27 15:11:56 on dist-test-slave-g0w1"
server_key: "7ecb0ffb60d4e1d179cb276ddc2b16c4"
server_key_iv: "2c6026e235ba5002f3de458ac48c31b8"
server_key_version: "encryptionkey@0"
I20260927 15:11:56.871842 21119 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 15:11:56.876075 21134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:56.877727 21119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:56.877976 21119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "1aac6f9eee0942588a71dad2c2dac07b"
format_stamp: "Formatted at 2026-09-27 15:11:56 on dist-test-slave-g0w1"
server_key: "7ecb0ffb60d4e1d179cb276ddc2b16c4"
server_key_iv: "2c6026e235ba5002f3de458ac48c31b8"
server_key_version: "encryptionkey@0"
I20260927 15:11:56.878235 21119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:56.898999 21119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:56.903734 21119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:56.904201 21119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:56.923899 21119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44171
I20260927 15:11:56.923899 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44171 every 8 connection(s)
I20260927 15:11:56.925451 21119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:56.928598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21119
I20260927 15:11:56.928797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:56.929273 388 external_mini_cluster.cc:1428] Setting key 54e125d14afecbfb53e10d47f6013cee
I20260927 15:11:56.929746 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:56.942519 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b: Bootstrap starting.
Sep 27 15:11:56 dist-test-slave-g0w1 krb5kdc[21106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:11:56.946455 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:56.947726 21186 log.cc:824] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:56.950757 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b: No bootstrap required, opened a new log
I20260927 15:11:56.952148 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.931090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33441 (local address 127.0.0.1:44171)
0927 15:11:56.931894 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.931907 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.931948 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:11:56.933521 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.933543 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.933974 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.934421 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:56.936010 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.937368 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.938716 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.939545 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.939759 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.944347 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:56.944358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:56.944374 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:56.944424 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:56.947772 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:56.948574 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:56.948584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:56.948593 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:56.948784 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:56.949185 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:56.949193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:56.949197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:56.949775 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:56.950094 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.950550 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.951129 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":375,"threads_started":1}
I20260927 15:11:56.956430 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:56.959784 21186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aac6f9eee0942588a71dad2c2dac07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44171 } }
I20260927 15:11:56.960227 21186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:56.960340 21186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aac6f9eee0942588a71dad2c2dac07b, State: Initialized, Role: FOLLOWER
I20260927 15:11:56.961112 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aac6f9eee0942588a71dad2c2dac07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44171 } }
I20260927 15:11:56.961439 21186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:56.961618 21186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:56.961833 21186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:56.964329 21186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aac6f9eee0942588a71dad2c2dac07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44171 } }
I20260927 15:11:56.964910 21186 leader_election.cc:302] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aac6f9eee0942588a71dad2c2dac07b; no voters:
I20260927 15:11:56.965672 21186 leader_election.cc:288] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:56.965780 21196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:56.967485 21196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [term 1 LEADER]: Becoming Leader. State: Replica: 1aac6f9eee0942588a71dad2c2dac07b, State: Running, Role: LEADER
I20260927 15:11:56.968210 21196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aac6f9eee0942588a71dad2c2dac07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44171 } }
I20260927 15:11:56.969624 21186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:56.977363 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.960934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57661 (local address 127.0.0.1:44171)
0927 15:11:56.961099 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.961104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.961636 (+ 532us) server_negotiation.cc:407] Connection header received
0927 15:11:56.961843 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.961847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.961944 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.962061 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:56.963792 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.964600 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.966082 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.967862 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.968670 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.970746 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:56.970754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:56.970764 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:56.970796 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:56.974232 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:56.974971 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:56.974981 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:56.974985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:56.975242 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:56.975692 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:56.975698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:56.975702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:56.976037 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:56.976350 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.976716 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.976969 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:11:56.984035 21198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aac6f9eee0942588a71dad2c2dac07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aac6f9eee0942588a71dad2c2dac07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44171 } } }
I20260927 15:11:56.983875 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aac6f9eee0942588a71dad2c2dac07b. Latest consensus state: current_term: 1 leader_uuid: "1aac6f9eee0942588a71dad2c2dac07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aac6f9eee0942588a71dad2c2dac07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44171 } } }
I20260927 15:11:56.984526 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:56.984396 21198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:56.986328 21211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:56.991503 21211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:57.000921 21211 catalog_manager.cc:1347] Generated new cluster ID: cb1ebb62303e46f88321c4588973c914
I20260927 15:11:57.001088 21211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:57.017676 21211 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:57.019244 21211 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:57.025693 21211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aac6f9eee0942588a71dad2c2dac07b: Generated new TSK 0
I20260927 15:11:57.026732 21211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:57.038497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21119
I20260927 15:11:57.040145 21181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:11:57.134876 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21119
2026-09-27T15:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2151 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:57 dist-test-slave-g0w1 krb5kdc[21218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:57 dist-test-slave-g0w1 krb5kdc[21218](info): set up 2 sockets
Sep 27 15:11:57 dist-test-slave-g0w1 krb5kdc[21218](info): commencing operation
krb5kdc: starting...
W20260927 15:11:58.247551 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:11:58 dist-test-slave-g0w1 krb5kdc[21218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:58.519141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:58.878155 21231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:58.878486 21231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:58.878687 21231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:58.886390 21231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:58.886520 21231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:58.886631 21231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:58.886724 21231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:58.886786 21231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:58.897250 21231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:11:58.898734 21231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:58.900188 21231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:58.912781 21239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:58.912856 21237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:58.912869 21236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:58.913753 21231 server_base.cc:1034] running on GCE node
I20260927 15:11:58.914768 21231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:58.916502 21231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:58.917766 21231 hybrid_clock.cc:648] HybridClock initialized: now 1790521918917723 us; error 44 us; skew 500 ppm
I20260927 15:11:58.918208 21231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:11:58 dist-test-slave-g0w1 krb5kdc[21218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:58.921912 21231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:58.923749 21231 webserver.cc:460] Webserver started at http://127.0.0.1:36077/ using document root <none> and password file <none>
I20260927 15:11:58.924598 21231 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:58.924772 21231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:58.925310 21231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:58.929023 21231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "02e13c8151744442abb1493fe5483a72"
format_stamp: "Formatted at 2026-09-27 15:11:58 on dist-test-slave-g0w1"
server_key: "17882c1b403f5db6954ed90e830c0a22"
server_key_iv: "f170e153f8d1decf4da625a50ad3a73b"
server_key_version: "encryptionkey@0"
I20260927 15:11:58.930016 21231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "02e13c8151744442abb1493fe5483a72"
format_stamp: "Formatted at 2026-09-27 15:11:58 on dist-test-slave-g0w1"
server_key: "17882c1b403f5db6954ed90e830c0a22"
server_key_iv: "f170e153f8d1decf4da625a50ad3a73b"
server_key_version: "encryptionkey@0"
I20260927 15:11:58.936352 21231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:11:58.940434 21246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:58.942224 21231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260927 15:11:58.942489 21231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "02e13c8151744442abb1493fe5483a72"
format_stamp: "Formatted at 2026-09-27 15:11:58 on dist-test-slave-g0w1"
server_key: "17882c1b403f5db6954ed90e830c0a22"
server_key_iv: "f170e153f8d1decf4da625a50ad3a73b"
server_key_version: "encryptionkey@0"
I20260927 15:11:58.942754 21231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:58.968032 21231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:58.973064 21231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:58.973380 21231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:58.993899 21231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39841
I20260927 15:11:58.993930 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39841 every 8 connection(s)
I20260927 15:11:58.995468 21231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:58.996758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21231
I20260927 15:11:58.996958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:11:58.997483 388 external_mini_cluster.cc:1428] Setting key 3da206316a15779cbf64f324a9262008
I20260927 15:11:59.000685 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:59.012287 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72: Bootstrap starting.
I20260927 15:11:59.012343 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:58.999486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48467 (local address 127.0.0.1:39841)
0927 15:11:59.000355 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:59.000369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:59.000424 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:11:59.002384 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:59.002416 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:59.002866 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:59.003388 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:59.004014 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:59.005539 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:59.007027 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:59.007907 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:59.008135 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:59.009545 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:59.009557 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:59.009579 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:59.010256 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:59.010349 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:59.010384 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:59.010546 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:59.010812 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 15:11:59.011224 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":394,"threads_started":1}
I20260927 15:11:59.015939 21298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:59.016630 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:59.017184 21298 log.cc:824] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:59.020254 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72: No bootstrap required, opened a new log
I20260927 15:11:59.029078 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:59.021715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33971 (local address 127.0.0.1:39841)
0927 15:11:59.021935 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:59.021943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:59.022344 (+ 401us) server_negotiation.cc:407] Connection header received
0927 15:11:59.022539 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:59.022545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:59.022629 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:59.022749 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:59.023347 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:59.024508 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:59.026006 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:59.026786 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:59.026932 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:59.028230 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:59.028237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:59.028243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:59.028267 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:59.028330 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:59.028356 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:59.028464 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:59.028609 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 15:11:59.028724 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:11:59.030366 21298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e13c8151744442abb1493fe5483a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39841 } }
I20260927 15:11:59.030930 21298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:59.031072 21298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e13c8151744442abb1493fe5483a72, State: Initialized, Role: FOLLOWER
I20260927 15:11:59.031713 21298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e13c8151744442abb1493fe5483a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39841 } }
I20260927 15:11:59.031965 21298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:59.032138 21298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:59.032346 21298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:59.034850 21298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e13c8151744442abb1493fe5483a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39841 } }
I20260927 15:11:59.035485 21298 leader_election.cc:302] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02e13c8151744442abb1493fe5483a72; no voters:
I20260927 15:11:59.036099 21298 leader_election.cc:288] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:59.036267 21308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:59.037842 21308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [term 1 LEADER]: Becoming Leader. State: Replica: 02e13c8151744442abb1493fe5483a72, State: Running, Role: LEADER
I20260927 15:11:59.038522 21308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e13c8151744442abb1493fe5483a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39841 } }
I20260927 15:11:59.038645 21298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:59.041692 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02e13c8151744442abb1493fe5483a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e13c8151744442abb1493fe5483a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39841 } } }
I20260927 15:11:59.041987 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02e13c8151744442abb1493fe5483a72. Latest consensus state: current_term: 1 leader_uuid: "02e13c8151744442abb1493fe5483a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e13c8151744442abb1493fe5483a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39841 } } }
I20260927 15:11:59.042244 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:59.042309 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:59.043773 21316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:59.049767 21316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:59.059432 21316 catalog_manager.cc:1347] Generated new cluster ID: 61b3ed2afc3b42e5bd097277ced65d04
I20260927 15:11:59.059581 21316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:59.075914 21316 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:59.077975 21316 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:59.086563 21316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02e13c8151744442abb1493fe5483a72: Generated new TSK 0
I20260927 15:11:59.087607 21316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:59.100071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21231
W20260927 15:11:59.101500 21293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48467
W20260927 15:11:59.102452 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:11:59.102583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21231
2026-09-27T15:11:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:59 dist-test-slave-g0w1 krb5kdc[21329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:59 dist-test-slave-g0w1 krb5kdc[21329](info): set up 2 sockets
Sep 27 15:11:59 dist-test-slave-g0w1 krb5kdc[21329](info): commencing operation
krb5kdc: starting...
W20260927 15:12:00.219632 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:00 dist-test-slave-g0w1 krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:00.495920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:00.848405 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:00.848739 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:00.848834 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:00.856453 21342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:00.856590 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:00.856638 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:00.856702 21342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:00.856746 21342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:00.867174 21342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:00.868566 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:00.869866 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:00.882827 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:00.882738 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:00.882823 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:00.883409 21342 server_base.cc:1034] running on GCE node
I20260927 15:12:00.884418 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:00.885907 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:00.887207 21342 hybrid_clock.cc:648] HybridClock initialized: now 1790521920887130 us; error 59 us; skew 500 ppm
I20260927 15:12:00.887559 21342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:00 dist-test-slave-g0w1 krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:00.891016 21342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:00.892652 21342 webserver.cc:460] Webserver started at http://127.0.0.1:33347/ using document root <none> and password file <none>
I20260927 15:12:00.893401 21342 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:00.893553 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:00.893875 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:00.897229 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "4ac4ea75314844a7b7a780deff7ca16d"
format_stamp: "Formatted at 2026-09-27 15:12:00 on dist-test-slave-g0w1"
server_key: "95a1d376af6fda13059c5b39106a1593"
server_key_iv: "198b52095f922c5a431627a608e1f7a6"
server_key_version: "encryptionkey@0"
I20260927 15:12:00.898104 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "4ac4ea75314844a7b7a780deff7ca16d"
format_stamp: "Formatted at 2026-09-27 15:12:00 on dist-test-slave-g0w1"
server_key: "95a1d376af6fda13059c5b39106a1593"
server_key_iv: "198b52095f922c5a431627a608e1f7a6"
server_key_version: "encryptionkey@0"
I20260927 15:12:00.904206 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 15:12:00.907903 21357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:00.909365 21342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:12:00.909582 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "4ac4ea75314844a7b7a780deff7ca16d"
format_stamp: "Formatted at 2026-09-27 15:12:00 on dist-test-slave-g0w1"
server_key: "95a1d376af6fda13059c5b39106a1593"
server_key_iv: "198b52095f922c5a431627a608e1f7a6"
server_key_version: "encryptionkey@0"
I20260927 15:12:00.909801 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:00.916204 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:00.920557 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:00.920871 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:00.940768 21342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41699
I20260927 15:12:00.940824 21408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41699 every 8 connection(s)
I20260927 15:12:00.942243 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:00.943709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21342
I20260927 15:12:00.943888 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:00.944415 388 external_mini_cluster.cc:1428] Setting key bf8bf95c8545f0392fb671133a403fb9
I20260927 15:12:00.947052 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:00.952745 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:00.946266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36633 (local address 127.0.0.1:41699)
0927 15:12:00.947169 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:00.947188 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:00.947248 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:12:00.948930 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:00.948960 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:00.949322 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:00.949717 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:00.950927 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:00.950934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:00.950947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:00.950961 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:00.951150 (+ 189us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:00.951180 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:00.951382 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:00.951511 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:12:00.951753 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260927 15:12:00.956873 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:00.960340 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d: Bootstrap starting.
I20260927 15:12:00.964869 21409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:00.965380 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:00.962591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49345 (local address 127.0.0.1:41699)
0927 15:12:00.962834 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:00.962840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:00.963174 (+ 334us) server_negotiation.cc:407] Connection header received
0927 15:12:00.963365 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:00.963369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:00.963421 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:00.963506 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:00.964582 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:00.964586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:00.964589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:00.964604 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:00.964645 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:00.964660 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:00.964764 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:00.964945 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 15:12:00.965084 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 15:12:00.966166 21409 log.cc:824] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:00.969147 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d: No bootstrap required, opened a new log
I20260927 15:12:00.977216 21409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac4ea75314844a7b7a780deff7ca16d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41699 } }
I20260927 15:12:00.977674 21409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:00.977797 21409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac4ea75314844a7b7a780deff7ca16d, State: Initialized, Role: FOLLOWER
I20260927 15:12:00.978448 21409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac4ea75314844a7b7a780deff7ca16d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41699 } }
I20260927 15:12:00.978693 21409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:00.978849 21409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:00.979046 21409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:00.981838 21409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac4ea75314844a7b7a780deff7ca16d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41699 } }
I20260927 15:12:00.982486 21409 leader_election.cc:302] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ac4ea75314844a7b7a780deff7ca16d; no voters:
I20260927 15:12:00.983201 21409 leader_election.cc:288] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:00.983399 21419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:00.984139 21419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [term 1 LEADER]: Becoming Leader. State: Replica: 4ac4ea75314844a7b7a780deff7ca16d, State: Running, Role: LEADER
I20260927 15:12:00.984740 21419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac4ea75314844a7b7a780deff7ca16d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41699 } }
I20260927 15:12:00.985460 21409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:00.987941 21420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ac4ea75314844a7b7a780deff7ca16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac4ea75314844a7b7a780deff7ca16d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41699 } } }
I20260927 15:12:00.988626 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ac4ea75314844a7b7a780deff7ca16d. Latest consensus state: current_term: 1 leader_uuid: "4ac4ea75314844a7b7a780deff7ca16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac4ea75314844a7b7a780deff7ca16d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41699 } } }
I20260927 15:12:00.988888 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:00.989519 21420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:00.992066 21428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:00.996658 21428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:01.005995 21428 catalog_manager.cc:1347] Generated new cluster ID: 303bc30119cf4978a856261dcdd9016d
I20260927 15:12:01.006142 21428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:01.017212 21428 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:01.017804 21428 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:01.038364 21428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ac4ea75314844a7b7a780deff7ca16d: Generated new TSK 0
I20260927 15:12:01.039690 21428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:01.052649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21342
W20260927 15:12:01.053938 21404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36633
W20260927 15:12:01.054745 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:01.054917 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21342
2026-09-27T15:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:01 dist-test-slave-g0w1 krb5kdc[21440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:01 dist-test-slave-g0w1 krb5kdc[21440](info): set up 2 sockets
Sep 27 15:12:01 dist-test-slave-g0w1 krb5kdc[21440](info): commencing operation
krb5kdc: starting...
W20260927 15:12:02.171643 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:02 dist-test-slave-g0w1 krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:02.450030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:02.799556 21453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:02.799969 21453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:02.800144 21453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:02.807719 21453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:02.807920 21453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:02.808058 21453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:02.808164 21453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:02.808255 21453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:02.818516 21453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:02.820042 21453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:02.821456 21453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:02.833748 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:02.833758 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:02.833781 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:02.835175 21453 server_base.cc:1034] running on GCE node
I20260927 15:12:02.836043 21453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:02.837630 21453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:02.838860 21453 hybrid_clock.cc:648] HybridClock initialized: now 1790521922838814 us; error 39 us; skew 500 ppm
I20260927 15:12:02.839359 21453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:02 dist-test-slave-g0w1 krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:02.842947 21453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:02.844759 21453 webserver.cc:460] Webserver started at http://127.0.0.1:43413/ using document root <none> and password file <none>
I20260927 15:12:02.845515 21453 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:02.845687 21453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:02.846025 21453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:02.849330 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "937eb331e6ce4651b803b36dafbcac7f"
format_stamp: "Formatted at 2026-09-27 15:12:02 on dist-test-slave-g0w1"
server_key: "284f96c1951530605f4072052a450903"
server_key_iv: "f7e9246d8f9ba60562cfc80417aff7d8"
server_key_version: "encryptionkey@0"
I20260927 15:12:02.850592 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "937eb331e6ce4651b803b36dafbcac7f"
format_stamp: "Formatted at 2026-09-27 15:12:02 on dist-test-slave-g0w1"
server_key: "284f96c1951530605f4072052a450903"
server_key_iv: "f7e9246d8f9ba60562cfc80417aff7d8"
server_key_version: "encryptionkey@0"
I20260927 15:12:02.856856 21453 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260927 15:12:02.860878 21468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:02.862564 21453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:12:02.862804 21453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "937eb331e6ce4651b803b36dafbcac7f"
format_stamp: "Formatted at 2026-09-27 15:12:02 on dist-test-slave-g0w1"
server_key: "284f96c1951530605f4072052a450903"
server_key_iv: "f7e9246d8f9ba60562cfc80417aff7d8"
server_key_version: "encryptionkey@0"
I20260927 15:12:02.863122 21453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:02.884531 21453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:02.889362 21453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:02.889717 21453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:02.909576 21453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35481
I20260927 15:12:02.909606 21519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35481 every 8 connection(s)
I20260927 15:12:02.911141 21453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:02.915023 21520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:02.917026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21453
I20260927 15:12:02.917225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:02.917743 388 external_mini_cluster.cc:1428] Setting key 0265bcebbf3f1a4a756a582f006f2329
I20260927 15:12:02.928974 21520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f: Bootstrap starting.
Sep 27 15:12:02 dist-test-slave-g0w1 krb5kdc[21440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:12:02.932538 21520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:02.933925 21520 log.cc:824] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:02.937821 21520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f: No bootstrap required, opened a new log
I20260927 15:12:02.942248 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:02.919766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:48711 (local address 127.0.0.1:35481)
0927 15:12:02.920608 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:02.920621 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:02.920666 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:12:02.922341 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:02.922365 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:02.922839 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:02.923305 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:02.924811 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:02.926160 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:02.927637 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:02.928372 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:02.928616 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:02.933207 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:02.933215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:02.933232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:02.933278 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:02.936951 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:02.937910 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:02.937922 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:02.937932 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:02.938135 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:02.938650 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:02.938654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:02.938656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:02.939273 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:02.939681 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:02.940249 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 15:12:02.940685 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":384,"threads_started":1}
I20260927 15:12:02.946861 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:02.948132 21520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937eb331e6ce4651b803b36dafbcac7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35481 } }
I20260927 15:12:02.948561 21520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:02.948695 21520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937eb331e6ce4651b803b36dafbcac7f, State: Initialized, Role: FOLLOWER
I20260927 15:12:02.949338 21520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937eb331e6ce4651b803b36dafbcac7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35481 } }
I20260927 15:12:02.949620 21520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:02.949836 21520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:02.950043 21520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:02.953609 21520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937eb331e6ce4651b803b36dafbcac7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35481 } }
I20260927 15:12:02.954310 21520 leader_election.cc:302] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937eb331e6ce4651b803b36dafbcac7f; no voters:
I20260927 15:12:02.955114 21520 leader_election.cc:288] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:02.955322 21530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:02.956449 21530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [term 1 LEADER]: Becoming Leader. State: Replica: 937eb331e6ce4651b803b36dafbcac7f, State: Running, Role: LEADER
I20260927 15:12:02.957126 21530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937eb331e6ce4651b803b36dafbcac7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35481 } }
I20260927 15:12:02.957700 21520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:02.960281 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937eb331e6ce4651b803b36dafbcac7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937eb331e6ce4651b803b36dafbcac7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35481 } } }
I20260927 15:12:02.960723 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:02.961324 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937eb331e6ce4651b803b36dafbcac7f. Latest consensus state: current_term: 1 leader_uuid: "937eb331e6ce4651b803b36dafbcac7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937eb331e6ce4651b803b36dafbcac7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35481 } } }
I20260927 15:12:02.961625 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:02.968417 21535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:02.971863 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:02.951837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:53927 (local address 127.0.0.1:35481)
0927 15:12:02.952087 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:02.952095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:02.952528 (+ 433us) server_negotiation.cc:407] Connection header received
0927 15:12:02.952662 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:02.952667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:02.952753 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:02.952863 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:02.954314 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:02.955570 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:02.957686 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:02.958381 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:02.958526 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:02.962249 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:02.962259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:02.962264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:02.962317 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:02.967275 (+ 4958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:02.968492 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:02.968510 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:02.968513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:02.968741 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:02.969462 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:02.969466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:02.969468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:02.969855 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:02.970182 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:02.970662 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 15:12:02.970820 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 15:12:02.977367 21535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:02.989379 21535 catalog_manager.cc:1347] Generated new cluster ID: c7c8c53d24de497db96e3142ca960a03
I20260927 15:12:02.989521 21535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:03.011672 21535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:03.013744 21535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:03.022396 21535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937eb331e6ce4651b803b36dafbcac7f: Generated new TSK 0
I20260927 15:12:03.023631 21535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:03.041555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21453
I20260927 15:12:03.042898 21515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:03.139780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21453
W20260927 15:12:03.159554 21438 connection.cc:535] client connection to 127.0.0.1:35481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35481): Connection reset by peer (error 104)
2026-09-27T15:12:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:03 dist-test-slave-g0w1 krb5kdc[21552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:03 dist-test-slave-g0w1 krb5kdc[21552](info): set up 2 sockets
Sep 27 15:12:03 dist-test-slave-g0w1 krb5kdc[21552](info): commencing operation
krb5kdc: starting...
W20260927 15:12:04.255712 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:04 dist-test-slave-g0w1 krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:04.530238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:04.881552 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:04.881961 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:04.882144 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:04.889683 21565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:04.889881 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:04.890013 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:04.890120 21565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:04.890211 21565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:04.900763 21565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:04.902225 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:04.903671 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:04.916509 21570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:04.916465 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:04.916544 21573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:04.917799 21565 server_base.cc:1034] running on GCE node
I20260927 15:12:04.918793 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:04.920746 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:04.921969 21565 hybrid_clock.cc:648] HybridClock initialized: now 1790521924921880 us; error 89 us; skew 500 ppm
I20260927 15:12:04.922361 21565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:04 dist-test-slave-g0w1 krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:04.925865 21565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:04.927578 21565 webserver.cc:460] Webserver started at http://127.0.0.1:39121/ using document root <none> and password file <none>
I20260927 15:12:04.928340 21565 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:04.928448 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:04.928805 21565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:04.932170 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "967992694d2c4c35b38bcf2304c509c5"
format_stamp: "Formatted at 2026-09-27 15:12:04 on dist-test-slave-g0w1"
server_key: "d7163393726559e6a09a8152150d7870"
server_key_iv: "24702ea21b5b2e53b63a55c483220079"
server_key_version: "encryptionkey@0"
I20260927 15:12:04.932987 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "967992694d2c4c35b38bcf2304c509c5"
format_stamp: "Formatted at 2026-09-27 15:12:04 on dist-test-slave-g0w1"
server_key: "d7163393726559e6a09a8152150d7870"
server_key_iv: "24702ea21b5b2e53b63a55c483220079"
server_key_version: "encryptionkey@0"
I20260927 15:12:04.938795 21565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 15:12:04.942747 21580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:04.944612 21565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:04.944832 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "967992694d2c4c35b38bcf2304c509c5"
format_stamp: "Formatted at 2026-09-27 15:12:04 on dist-test-slave-g0w1"
server_key: "d7163393726559e6a09a8152150d7870"
server_key_iv: "24702ea21b5b2e53b63a55c483220079"
server_key_version: "encryptionkey@0"
I20260927 15:12:04.945115 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:04.965616 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:04.970628 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:04.970952 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:04.991523 21565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43489
I20260927 15:12:04.991524 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43489 every 8 connection(s)
I20260927 15:12:04.993028 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:04.997145 21632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:04.997363 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21565
I20260927 15:12:04.997532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:04.998090 388 external_mini_cluster.cc:1428] Setting key fd3c19b9584f73cc8ab0ab783f27525a
I20260927 15:12:05.011783 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5: Bootstrap starting.
I20260927 15:12:05.012452 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:05.000182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:56479 (local address 127.0.0.1:43489)
0927 15:12:05.001065 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:05.001078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:05.001125 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:12:05.002799 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:05.002825 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:05.003470 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:05.003919 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:05.004529 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:05.005864 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:05.007321 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:05.008055 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:05.008289 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:05.009780 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:05.009789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:05.009806 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:05.010427 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:05.010513 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:05.010542 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:05.010731 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:05.011125 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 15:12:05.011329 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":379,"threads_started":1}
I20260927 15:12:05.015357 21632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:05.016753 21632 log.cc:824] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:05.016804 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:05.020138 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5: No bootstrap required, opened a new log
I20260927 15:12:05.028191 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:05.021342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:43207 (local address 127.0.0.1:43489)
0927 15:12:05.021510 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:05.021534 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:05.022012 (+ 478us) server_negotiation.cc:407] Connection header received
0927 15:12:05.022265 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:05.022271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:05.022346 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:05.022465 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:05.023058 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:05.023905 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:05.025355 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:05.025854 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:05.026001 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:05.027376 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:05.027380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:05.027383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:05.027407 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:05.027457 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:05.027474 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:05.027613 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:05.027816 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 15:12:05.027881 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:12:05.029095 21632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967992694d2c4c35b38bcf2304c509c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260927 15:12:05.029613 21632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:05.029762 21632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967992694d2c4c35b38bcf2304c509c5, State: Initialized, Role: FOLLOWER
I20260927 15:12:05.030436 21632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967992694d2c4c35b38bcf2304c509c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260927 15:12:05.030681 21632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:05.030789 21632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:05.030938 21632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:05.033372 21632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967992694d2c4c35b38bcf2304c509c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260927 15:12:05.033941 21632 leader_election.cc:302] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 967992694d2c4c35b38bcf2304c509c5; no voters:
I20260927 15:12:05.034647 21632 leader_election.cc:288] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:05.034767 21642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:05.036051 21642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [term 1 LEADER]: Becoming Leader. State: Replica: 967992694d2c4c35b38bcf2304c509c5, State: Running, Role: LEADER
I20260927 15:12:05.036684 21642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967992694d2c4c35b38bcf2304c509c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260927 15:12:05.037153 21632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:05.040316 21644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967992694d2c4c35b38bcf2304c509c5. Latest consensus state: current_term: 1 leader_uuid: "967992694d2c4c35b38bcf2304c509c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967992694d2c4c35b38bcf2304c509c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } } }
I20260927 15:12:05.040788 21644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:05.041352 21643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "967992694d2c4c35b38bcf2304c509c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967992694d2c4c35b38bcf2304c509c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } } }
I20260927 15:12:05.041667 21643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:05.042825 21647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:05.049160 21647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:05.059604 21647 catalog_manager.cc:1347] Generated new cluster ID: 444c8c54ea7d44b48b84cdae2c2ea142
I20260927 15:12:05.059751 21647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:05.067778 21647 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:05.069804 21647 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:05.083339 21647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 967992694d2c4c35b38bcf2304c509c5: Generated new TSK 0
I20260927 15:12:05.084380 21647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:05.100179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21565
W20260927 15:12:05.102008 21627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:56479
W20260927 15:12:05.102897 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:05.103133 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21565
W20260927 15:12:05.120257 21550 connection.cc:535] client connection to 127.0.0.1:43489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43489): Connection reset by peer (error 104)
2026-09-27T15:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:05 dist-test-slave-g0w1 krb5kdc[21663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:05 dist-test-slave-g0w1 krb5kdc[21663](info): set up 2 sockets
Sep 27 15:12:05 dist-test-slave-g0w1 krb5kdc[21663](info): commencing operation
krb5kdc: starting...
W20260927 15:12:06.235726 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:06 dist-test-slave-g0w1 krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:06.510829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:06.860059 21676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:06.860405 21676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:06.860584 21676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:06.868397 21676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:06.868534 21676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:06.868646 21676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:06.868757 21676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:06.868839 21676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:06.879573 21676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:06.881011 21676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:06.882424 21676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:06.895008 21681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:06.895157 21684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:06.895123 21682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:06.896169 21676 server_base.cc:1034] running on GCE node
I20260927 15:12:06.897078 21676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:06.898696 21676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:06.899953 21676 hybrid_clock.cc:648] HybridClock initialized: now 1790521926899906 us; error 46 us; skew 500 ppm
I20260927 15:12:06.900408 21676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:06 dist-test-slave-g0w1 krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:06.908623 21676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:06.910351 21676 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260927 15:12:06.911185 21676 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:06.911302 21676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:06.911590 21676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:06.914773 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "91e876f652d94461b1989dd559b4d7a1"
format_stamp: "Formatted at 2026-09-27 15:12:06 on dist-test-slave-g0w1"
server_key: "478761af271e17e7d4568a8076ee87f7"
server_key_iv: "55279ef2ff52ccc9ad03b7820e8f2b48"
server_key_version: "encryptionkey@0"
I20260927 15:12:06.915635 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "91e876f652d94461b1989dd559b4d7a1"
format_stamp: "Formatted at 2026-09-27 15:12:06 on dist-test-slave-g0w1"
server_key: "478761af271e17e7d4568a8076ee87f7"
server_key_iv: "55279ef2ff52ccc9ad03b7820e8f2b48"
server_key_version: "encryptionkey@0"
I20260927 15:12:06.921545 21676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260927 15:12:06.925498 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:06.927274 21676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:06.927439 21676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "91e876f652d94461b1989dd559b4d7a1"
format_stamp: "Formatted at 2026-09-27 15:12:06 on dist-test-slave-g0w1"
server_key: "478761af271e17e7d4568a8076ee87f7"
server_key_iv: "55279ef2ff52ccc9ad03b7820e8f2b48"
server_key_version: "encryptionkey@0"
I20260927 15:12:06.927641 21676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:06.933879 21676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:06.938635 21676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:06.938901 21676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:06.959275 21676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34265
I20260927 15:12:06.959278 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34265 every 8 connection(s)
I20260927 15:12:06.960866 21676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:06.965109 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:06.967974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21676
I20260927 15:12:06.968154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:06.968719 388 external_mini_cluster.cc:1428] Setting key 6dad4b850d343dcdfe7ca0aa5cc4addd
I20260927 15:12:06.977470 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:06.970769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:34239 (local address 127.0.0.1:34265)
0927 15:12:06.971644 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:06.971658 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:06.971711 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:12:06.973584 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:06.973610 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:06.973996 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:06.974468 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:06.975659 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:06.975669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:06.975684 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:06.975710 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:06.975769 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:06.975792 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:06.976082 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:06.976243 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:12:06.976452 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":379,"threads_started":1}
I20260927 15:12:06.978377 21743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1: Bootstrap starting.
I20260927 15:12:06.982231 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:06.982594 21743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:06.983757 21743 log.cc:824] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:06.987010 21743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1: No bootstrap required, opened a new log
I20260927 15:12:06.990136 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:06.987285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:44803 (local address 127.0.0.1:34265)
0927 15:12:06.987501 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:06.987510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:06.987927 (+ 417us) server_negotiation.cc:407] Connection header received
0927 15:12:06.988138 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:06.988145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:06.988203 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:06.988313 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:06.989355 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:06.989360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:06.989375 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:06.989393 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:06.989438 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:06.989454 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:06.989642 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:06.989734 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:12:06.989814 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:12:06.996315 21743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e876f652d94461b1989dd559b4d7a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260927 15:12:06.996791 21743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:06.996909 21743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e876f652d94461b1989dd559b4d7a1, State: Initialized, Role: FOLLOWER
I20260927 15:12:06.997694 21743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e876f652d94461b1989dd559b4d7a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260927 15:12:06.998018 21743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:06.998168 21743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:06.998379 21743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:07.000824 21743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e876f652d94461b1989dd559b4d7a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260927 15:12:07.001392 21743 leader_election.cc:302] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91e876f652d94461b1989dd559b4d7a1; no voters:
I20260927 15:12:07.002063 21743 leader_election.cc:288] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:07.002249 21753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:07.003028 21753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [term 1 LEADER]: Becoming Leader. State: Replica: 91e876f652d94461b1989dd559b4d7a1, State: Running, Role: LEADER
I20260927 15:12:07.003657 21753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e876f652d94461b1989dd559b4d7a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260927 15:12:07.004354 21743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:07.006390 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91e876f652d94461b1989dd559b4d7a1. Latest consensus state: current_term: 1 leader_uuid: "91e876f652d94461b1989dd559b4d7a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e876f652d94461b1989dd559b4d7a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260927 15:12:07.006834 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:07.006717 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91e876f652d94461b1989dd559b4d7a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91e876f652d94461b1989dd559b4d7a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260927 15:12:07.007119 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:07.010078 21758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:07.017030 21758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:07.027812 21758 catalog_manager.cc:1347] Generated new cluster ID: 6752691f15e94c9fa24ec3e36735443d
I20260927 15:12:07.027977 21758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:07.050469 21758 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:07.051156 21758 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:07.067065 21758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91e876f652d94461b1989dd559b4d7a1: Generated new TSK 0
I20260927 15:12:07.068092 21758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:07.083701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21676
W20260927 15:12:07.085218 21738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:34239
W20260927 15:12:07.086007 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:07.086191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21676
2026-09-27T15:12:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:07 dist-test-slave-g0w1 krb5kdc[21774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:07 dist-test-slave-g0w1 krb5kdc[21774](info): set up 2 sockets
Sep 27 15:12:07 dist-test-slave-g0w1 krb5kdc[21774](info): commencing operation
krb5kdc: starting...
W20260927 15:12:08.223821 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:08 dist-test-slave-g0w1 krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:08.495819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:08.850829 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:08.851290 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:08.851470 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:08.859306 21787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:08.859452 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:08.859555 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:08.859663 21787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:08.859730 21787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:08.870188 21787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:08.871628 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:08.873018 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:08.885793 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:08.885738 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:08.885998 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:08.886626 21787 server_base.cc:1034] running on GCE node
I20260927 15:12:08.887591 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:08.889180 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:08.890429 21787 hybrid_clock.cc:648] HybridClock initialized: now 1790521928890378 us; error 36 us; skew 500 ppm
I20260927 15:12:08.890872 21787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:08 dist-test-slave-g0w1 krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:08.894413 21787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:08.896121 21787 webserver.cc:460] Webserver started at http://127.0.0.1:33593/ using document root <none> and password file <none>
I20260927 15:12:08.896872 21787 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:08.897045 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:08.897383 21787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:08.900681 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "aff3e03db35c45cab19af247c5c974dd"
format_stamp: "Formatted at 2026-09-27 15:12:08 on dist-test-slave-g0w1"
server_key: "ac9961eeabf1ba3c675518d69d07d4f5"
server_key_iv: "9dbbde99259fa7eb7381b38d03b8e36d"
server_key_version: "encryptionkey@0"
I20260927 15:12:08.901494 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "aff3e03db35c45cab19af247c5c974dd"
format_stamp: "Formatted at 2026-09-27 15:12:08 on dist-test-slave-g0w1"
server_key: "ac9961eeabf1ba3c675518d69d07d4f5"
server_key_iv: "9dbbde99259fa7eb7381b38d03b8e36d"
server_key_version: "encryptionkey@0"
I20260927 15:12:08.907030 21787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:12:08.910988 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:08.912730 21787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:08.912966 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "aff3e03db35c45cab19af247c5c974dd"
format_stamp: "Formatted at 2026-09-27 15:12:08 on dist-test-slave-g0w1"
server_key: "ac9961eeabf1ba3c675518d69d07d4f5"
server_key_iv: "9dbbde99259fa7eb7381b38d03b8e36d"
server_key_version: "encryptionkey@0"
I20260927 15:12:08.913201 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:08.934103 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:08.938824 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:08.939250 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:08.959314 21787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37271
I20260927 15:12:08.959337 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37271 every 8 connection(s)
I20260927 15:12:08.960855 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:08.963219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21787
I20260927 15:12:08.963442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:08.963963 388 external_mini_cluster.cc:1428] Setting key 86b34bc481db90164d7f32fcb72dfedf
I20260927 15:12:08.965652 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:12:08 dist-test-slave-g0w1 krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:12:08.980847 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd: Bootstrap starting.
I20260927 15:12:08.985507 21854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:08.986886 21854 log.cc:824] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:08.990717 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd: No bootstrap required, opened a new log
I20260927 15:12:08.990844 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:08.965976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:51175 (local address 127.0.0.1:37271)
0927 15:12:08.966760 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:08.966773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:08.966812 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:12:08.968534 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:08.968558 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:08.968988 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:08.969348 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:08.971234 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:08.972662 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:08.974534 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:08.975435 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:08.975594 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:08.980961 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:08.980975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:08.981000 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:08.981070 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:08.985380 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:08.986551 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:08.986562 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:08.986571 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:08.986836 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:08.987438 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:08.987445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:08.987452 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:08.988143 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:08.988442 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:08.989050 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:12:08.989416 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":352,"threads_started":1}
I20260927 15:12:08.995437 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:09.001544 21854 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3e03db35c45cab19af247c5c974dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 15:12:09.002004 21854 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:09.002142 21854 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff3e03db35c45cab19af247c5c974dd, State: Initialized, Role: FOLLOWER
I20260927 15:12:09.002952 21854 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3e03db35c45cab19af247c5c974dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 15:12:09.003312 21854 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:09.003470 21854 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:09.003683 21854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:09.006784 21854 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3e03db35c45cab19af247c5c974dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 15:12:09.007463 21854 leader_election.cc:302] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff3e03db35c45cab19af247c5c974dd; no voters:
I20260927 15:12:09.008275 21854 leader_election.cc:288] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:09.008387 21864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:09.011368 21854 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:09.011374 21864 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [term 1 LEADER]: Becoming Leader. State: Replica: aff3e03db35c45cab19af247c5c974dd, State: Running, Role: LEADER
I20260927 15:12:09.012239 21864 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3e03db35c45cab19af247c5c974dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 15:12:09.016582 21866 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff3e03db35c45cab19af247c5c974dd. Latest consensus state: current_term: 1 leader_uuid: "aff3e03db35c45cab19af247c5c974dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3e03db35c45cab19af247c5c974dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260927 15:12:09.017105 21866 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:09.017385 21865 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff3e03db35c45cab19af247c5c974dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3e03db35c45cab19af247c5c974dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260927 15:12:09.017747 21865 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:09.022007 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:09.001271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:49259 (local address 127.0.0.1:37271)
0927 15:12:09.001466 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:09.001474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:09.002102 (+ 628us) server_negotiation.cc:407] Connection header received
0927 15:12:09.002342 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:09.002347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:09.002439 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:09.002563 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:09.004736 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:09.005584 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:09.007457 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:09.007973 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:09.008125 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:09.011024 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:09.011033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:09.011040 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:09.011137 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:09.015746 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:09.017009 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:09.017014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:09.017018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:09.017864 (+ 846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:09.018787 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:09.018792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:09.018795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:09.019322 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:09.020947 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:09.021200 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:12:09.021468 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:12:09.026784 21874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:09.032480 21874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:09.042793 21874 catalog_manager.cc:1347] Generated new cluster ID: 68c19add58f8440fbe334b7cf49e9105
I20260927 15:12:09.042943 21874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:09.063738 21874 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:09.065351 21874 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:09.076767 21874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff3e03db35c45cab19af247c5c974dd: Generated new TSK 0
I20260927 15:12:09.077992 21874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:09.091174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21787
I20260927 15:12:09.092327 21849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:09.189298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21787
W20260927 15:12:09.207041 21772 connection.cc:535] client connection to 127.0.0.1:37271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37271): Connection reset by peer (error 104)
2026-09-27T15:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2103 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:09 dist-test-slave-g0w1 krb5kdc[21887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:09 dist-test-slave-g0w1 krb5kdc[21887](info): set up 2 sockets
Sep 27 15:12:09 dist-test-slave-g0w1 krb5kdc[21887](info): commencing operation
krb5kdc: starting...
W20260927 15:12:10.307619 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:10 dist-test-slave-g0w1 krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:10.578923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:10.934293 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:10.934612 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:10.934706 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:10.942238 21900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:10.942375 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:10.942423 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:10.942477 21900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:10.942520 21900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:10.952874 21900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:10.954223 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:10.955772 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:10.969054 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:10.969129 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:10.969007 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:10.969962 21900 server_base.cc:1034] running on GCE node
I20260927 15:12:10.970891 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:10.972574 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:10.973850 21900 hybrid_clock.cc:648] HybridClock initialized: now 1790521930973795 us; error 51 us; skew 500 ppm
I20260927 15:12:10.974313 21900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:10 dist-test-slave-g0w1 krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:10.977914 21900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:10.979894 21900 webserver.cc:460] Webserver started at http://127.0.0.1:38433/ using document root <none> and password file <none>
I20260927 15:12:10.980693 21900 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:10.980855 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:10.981213 21900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:10.984752 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "2447284ab81e4489b73fc22d0da42576"
format_stamp: "Formatted at 2026-09-27 15:12:10 on dist-test-slave-g0w1"
server_key: "22d7a2aa440c014069f6240dac12b183"
server_key_iv: "fb2067db9652e8d0fae6a0361aa6d077"
server_key_version: "encryptionkey@0"
I20260927 15:12:10.985672 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "2447284ab81e4489b73fc22d0da42576"
format_stamp: "Formatted at 2026-09-27 15:12:10 on dist-test-slave-g0w1"
server_key: "22d7a2aa440c014069f6240dac12b183"
server_key_iv: "fb2067db9652e8d0fae6a0361aa6d077"
server_key_version: "encryptionkey@0"
I20260927 15:12:10.991668 21900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:12:10.995883 21915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:10.997586 21900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:12:10.997828 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "2447284ab81e4489b73fc22d0da42576"
format_stamp: "Formatted at 2026-09-27 15:12:10 on dist-test-slave-g0w1"
server_key: "22d7a2aa440c014069f6240dac12b183"
server_key_iv: "fb2067db9652e8d0fae6a0361aa6d077"
server_key_version: "encryptionkey@0"
I20260927 15:12:10.998073 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:11.014868 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:11.019740 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:11.020078 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:11.040221 21900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35885
I20260927 15:12:11.040271 21966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35885 every 8 connection(s)
I20260927 15:12:11.041740 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:11.046392 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:11.046487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 21900
I20260927 15:12:11.046648 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:11.047233 388 external_mini_cluster.cc:1428] Setting key 08fd88806e262b6a43dc0e2786389ba9
I20260927 15:12:11.061769 21967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576: Bootstrap starting.
I20260927 15:12:11.065331 21967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:11.066660 21967 log.cc:824] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:11.067783 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:11.055214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:34533 (local address 127.0.0.1:35885)
0927 15:12:11.056064 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:11.056077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:11.056124 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:12:11.057886 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:11.057914 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:11.058424 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:11.058887 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:11.059588 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:11.061010 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:11.062508 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:11.063312 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:11.063537 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:11.064866 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:11.064874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:11.064889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:11.065495 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:11.065602 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:11.065636 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:11.065835 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:11.066122 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:12:11.066309 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":368,"threads_started":1}
I20260927 15:12:11.070073 21967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576: No bootstrap required, opened a new log
I20260927 15:12:11.072103 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:11.078888 21967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2447284ab81e4489b73fc22d0da42576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260927 15:12:11.079456 21967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:11.079587 21967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2447284ab81e4489b73fc22d0da42576, State: Initialized, Role: FOLLOWER
I20260927 15:12:11.080334 21967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2447284ab81e4489b73fc22d0da42576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260927 15:12:11.080632 21967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:11.080781 21967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:11.081012 21967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:11.083895 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:11.076529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:52751 (local address 127.0.0.1:35885)
0927 15:12:11.076761 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:11.076770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:11.077130 (+ 360us) server_negotiation.cc:407] Connection header received
0927 15:12:11.077329 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:11.077333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:11.077404 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:11.077500 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:11.078145 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:11.078921 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:11.080911 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:11.081462 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:11.081590 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:11.082926 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:11.082931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:11.082934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:11.082957 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:11.083030 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:11.083051 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:11.083262 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:11.083474 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 15:12:11.083548 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:12:11.084373 21967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2447284ab81e4489b73fc22d0da42576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260927 15:12:11.085026 21967 leader_election.cc:302] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2447284ab81e4489b73fc22d0da42576; no voters:
I20260927 15:12:11.085731 21967 leader_election.cc:288] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:11.085863 21977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:11.086691 21977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [term 1 LEADER]: Becoming Leader. State: Replica: 2447284ab81e4489b73fc22d0da42576, State: Running, Role: LEADER
I20260927 15:12:11.087337 21977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2447284ab81e4489b73fc22d0da42576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260927 15:12:11.088150 21967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:11.090504 21978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2447284ab81e4489b73fc22d0da42576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2447284ab81e4489b73fc22d0da42576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260927 15:12:11.091167 21978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:11.095721 21979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2447284ab81e4489b73fc22d0da42576. Latest consensus state: current_term: 1 leader_uuid: "2447284ab81e4489b73fc22d0da42576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2447284ab81e4489b73fc22d0da42576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260927 15:12:11.096190 21979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:11.096585 21986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:11.102232 21986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:11.112133 21986 catalog_manager.cc:1347] Generated new cluster ID: 6640874ea9154eb99db571dea7fab494
I20260927 15:12:11.112268 21986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:11.160902 21986 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:11.162485 21986 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:11.173086 21986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2447284ab81e4489b73fc22d0da42576: Generated new TSK 0
I20260927 15:12:11.174314 21986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:11.189211 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21900
W20260927 15:12:11.190725 21962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:34533
W20260927 15:12:11.191715 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 21900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:11.191880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 21900
W20260927 15:12:11.209635 21885 connection.cc:535] client connection to 127.0.0.1:35885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35885): Connection reset by peer (error 104)
2026-09-27T15:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:11 dist-test-slave-g0w1 krb5kdc[21998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:11 dist-test-slave-g0w1 krb5kdc[21998](info): set up 2 sockets
Sep 27 15:12:11 dist-test-slave-g0w1 krb5kdc[21998](info): commencing operation
krb5kdc: starting...
W20260927 15:12:12.319761 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:12 dist-test-slave-g0w1 krb5kdc[21998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:12:12.587543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:12.934228 22011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:12.934530 22011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:12.934715 22011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:12.942338 22011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:12.942534 22011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:12.942665 22011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:12.942780 22011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:12.942860 22011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:12.953190 22011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:12.954660 22011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:12.956202 22011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:12.969007 22016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:12.968976 22019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:12.969059 22017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:12.969915 22011 server_base.cc:1034] running on GCE node
I20260927 15:12:12.970844 22011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:12.972465 22011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:12.973730 22011 hybrid_clock.cc:648] HybridClock initialized: now 1790521932973676 us; error 75 us; skew 500 ppm
I20260927 15:12:12.974120 22011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:12 dist-test-slave-g0w1 krb5kdc[21998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:12.985248 22011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:12:12.986930 22011 webserver.cc:460] Webserver started at http://127.0.0.1:33483/ using document root <none> and password file <none>
I20260927 15:12:12.987753 22011 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:12.987864 22011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:12.988130 22011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:12.991355 22011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "d7c50569a1ba4c0c8b91f33c5406f026"
format_stamp: "Formatted at 2026-09-27 15:12:12 on dist-test-slave-g0w1"
server_key: "9f39f02661f57f48e234d318928b8f2a"
server_key_iv: "05e776ccc637e857b797f61805fb018a"
server_key_version: "encryptionkey@0"
I20260927 15:12:12.992121 22011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7c50569a1ba4c0c8b91f33c5406f026"
format_stamp: "Formatted at 2026-09-27 15:12:12 on dist-test-slave-g0w1"
server_key: "9f39f02661f57f48e234d318928b8f2a"
server_key_iv: "05e776ccc637e857b797f61805fb018a"
server_key_version: "encryptionkey@0"
I20260927 15:12:12.998122 22011 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:12:13.002182 22026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:13.003981 22011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:13.004194 22011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d7c50569a1ba4c0c8b91f33c5406f026"
format_stamp: "Formatted at 2026-09-27 15:12:12 on dist-test-slave-g0w1"
server_key: "9f39f02661f57f48e234d318928b8f2a"
server_key_iv: "05e776ccc637e857b797f61805fb018a"
server_key_version: "encryptionkey@0"
I20260927 15:12:13.004388 22011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:13.010710 22011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:13.016052 22011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:13.016335 22011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:13.036785 22011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32821
I20260927 15:12:13.036826 22077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32821 every 8 connection(s)
I20260927 15:12:13.038314 22011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:13.042393 22078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:13.044672 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22011
I20260927 15:12:13.044868 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:13.045338 388 external_mini_cluster.cc:1428] Setting key b513da0c4bdf5562c81ef932b8a1a500
I20260927 15:12:13.053925 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:13.047394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:39297 (local address 127.0.0.1:32821)
0927 15:12:13.048289 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:13.048302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:13.048352 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:12:13.050047 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:13.050072 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:13.050491 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:13.050946 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:13.052104 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:13.052113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:13.052128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:13.052152 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:13.052208 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:13.052231 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:13.052498 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:13.052653 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 15:12:13.052855 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":379,"threads_started":1}
I20260927 15:12:13.055678 22078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026: Bootstrap starting.
I20260927 15:12:13.058396 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:13.059345 22078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:13.060489 22078 log.cc:824] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:13.063859 22078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026: No bootstrap required, opened a new log
I20260927 15:12:13.065949 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:13.063240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:53597 (local address 127.0.0.1:32821)
0927 15:12:13.063432 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:13.063440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:13.063872 (+ 432us) server_negotiation.cc:407] Connection header received
0927 15:12:13.064032 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:13.064037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:13.064101 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:13.064219 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:13.065206 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:13.065211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:13.065216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:13.065237 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:13.065289 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:13.065311 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:13.065454 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:13.065590 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:12:13.065657 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:12:13.072911 22078 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260927 15:12:13.073341 22078 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:13.073418 22078 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c50569a1ba4c0c8b91f33c5406f026, State: Initialized, Role: FOLLOWER
I20260927 15:12:13.074028 22078 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260927 15:12:13.074245 22078 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:13.074402 22078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:13.074607 22078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:13.077252 22078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260927 15:12:13.077811 22078 leader_election.cc:302] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7c50569a1ba4c0c8b91f33c5406f026; no voters:
I20260927 15:12:13.078487 22078 leader_election.cc:288] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:13.078670 22088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:13.079402 22088 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [term 1 LEADER]: Becoming Leader. State: Replica: d7c50569a1ba4c0c8b91f33c5406f026, State: Running, Role: LEADER
I20260927 15:12:13.080027 22088 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260927 15:12:13.081027 22078 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:13.083406 22090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c50569a1ba4c0c8b91f33c5406f026. Latest consensus state: current_term: 1 leader_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } } }
I20260927 15:12:13.083943 22090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:13.084144 22089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c50569a1ba4c0c8b91f33c5406f026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } } }
I20260927 15:12:13.084537 22089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:13.085634 22096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:13.091030 22096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:13.101500 22096 catalog_manager.cc:1347] Generated new cluster ID: d78b82517d014327b6e03976d2e1d92a
I20260927 15:12:13.101635 22096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:13.131235 22096 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:13.131718 22096 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:13.146797 22096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7c50569a1ba4c0c8b91f33c5406f026: Generated new TSK 0
I20260927 15:12:13.147861 22096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:13.157465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22011
W20260927 15:12:13.158666 22073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:39297
W20260927 15:12:13.159489 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22011: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:13.159669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22011
2026-09-27T15:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:13 dist-test-slave-g0w1 krb5kdc[22109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:13 dist-test-slave-g0w1 krb5kdc[22109](info): set up 2 sockets
Sep 27 15:12:13 dist-test-slave-g0w1 krb5kdc[22109](info): commencing operation
krb5kdc: starting...
W20260927 15:12:14.283511 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:14 dist-test-slave-g0w1 krb5kdc[22109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:14.554558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43417
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:14.894845 22122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:14.895293 22122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:14.895458 22122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:14.903174 22122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:14.903357 22122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:14.903479 22122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:14.903590 22122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:14.903671 22122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:14.913941 22122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:14.915398 22122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:14.916814 22122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:14.929528 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:14.929478 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:14.929569 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:14.930341 22122 server_base.cc:1034] running on GCE node
I20260927 15:12:14.931313 22122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:14.932879 22122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:14.934120 22122 hybrid_clock.cc:648] HybridClock initialized: now 1790521934934066 us; error 45 us; skew 500 ppm
I20260927 15:12:14.934515 22122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:14 dist-test-slave-g0w1 krb5kdc[22109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:14.938037 22122 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:14.939790 22122 webserver.cc:460] Webserver started at http://127.0.97.62:41451/ using document root <none> and password file <none>
I20260927 15:12:14.940551 22122 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:14.940711 22122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:14.941035 22122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:14.944319 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "db02c98c41614882990cc301e681e412"
format_stamp: "Formatted at 2026-09-27 15:12:14 on dist-test-slave-g0w1"
server_key: "3c65361783dc5dd49c07373aae2b23b3"
server_key_iv: "ea9eebeef252eac4df0c914fff275402"
server_key_version: "encryptionkey@0"
I20260927 15:12:14.945142 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "db02c98c41614882990cc301e681e412"
format_stamp: "Formatted at 2026-09-27 15:12:14 on dist-test-slave-g0w1"
server_key: "3c65361783dc5dd49c07373aae2b23b3"
server_key_iv: "ea9eebeef252eac4df0c914fff275402"
server_key_version: "encryptionkey@0"
I20260927 15:12:14.951958 22122 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:12:14.956112 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:14.957693 22122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260927 15:12:14.957922 22122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "db02c98c41614882990cc301e681e412"
format_stamp: "Formatted at 2026-09-27 15:12:14 on dist-test-slave-g0w1"
server_key: "3c65361783dc5dd49c07373aae2b23b3"
server_key_iv: "ea9eebeef252eac4df0c914fff275402"
server_key_version: "encryptionkey@0"
I20260927 15:12:14.958159 22122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:14.980984 22122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:14.986112 22122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:14.986410 22122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:15.006330 22122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43417
I20260927 15:12:15.006374 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43417 every 8 connection(s)
I20260927 15:12:15.007872 22122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:15.011475 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22122
I20260927 15:12:15.011689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:15.012117 388 external_mini_cluster.cc:1428] Setting key 164f1c3da9f677feb62d1d1084010999
I20260927 15:12:15.012142 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:15.027132 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412: Bootstrap starting.
Sep 27 15:12:15 dist-test-slave-g0w1 krb5kdc[22109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:15.031175 22189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:15.032290 22189 log.cc:824] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:15.035419 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412: No bootstrap required, opened a new log
I20260927 15:12:15.037457 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:15.013890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46821 (local address 127.0.97.62:43417)
0927 15:12:15.014681 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:15.014699 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:15.014757 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:12:15.017231 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:15.017265 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:15.017869 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:15.018412 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:15.019937 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:15.021963 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:15.023501 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:15.024649 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:15.024843 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:15.028990 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:15.029003 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:15.029023 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:15.029084 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:15.032963 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:15.033878 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:15.033885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:15.033892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:15.034074 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:15.034571 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:15.034575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:15.034577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:15.035180 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:15.035544 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:15.036069 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 15:12:15.036409 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":330,"threads_started":1}
I20260927 15:12:15.041826 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:15.044456 22189 raft_consensus.cc:348] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db02c98c41614882990cc301e681e412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43417 } }
I20260927 15:12:15.045013 22189 raft_consensus.cc:374] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:15.045146 22189 raft_consensus.cc:729] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db02c98c41614882990cc301e681e412, State: Initialized, Role: FOLLOWER
I20260927 15:12:15.045841 22189 consensus_queue.cc:260] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db02c98c41614882990cc301e681e412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43417 } }
I20260927 15:12:15.046186 22189 raft_consensus.cc:388] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:15.046361 22189 raft_consensus.cc:482] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:15.046612 22189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:15.049149 22189 raft_consensus.cc:504] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db02c98c41614882990cc301e681e412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43417 } }
I20260927 15:12:15.049733 22189 leader_election.cc:302] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db02c98c41614882990cc301e681e412; no voters:
I20260927 15:12:15.050509 22189 leader_election.cc:288] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:15.050647 22199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:15.051378 22199 raft_consensus.cc:686] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [term 1 LEADER]: Becoming Leader. State: Replica: db02c98c41614882990cc301e681e412, State: Running, Role: LEADER
I20260927 15:12:15.052085 22199 consensus_queue.cc:237] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db02c98c41614882990cc301e681e412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43417 } }
I20260927 15:12:15.054427 22189 sys_catalog.cc:564] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:15.055522 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db02c98c41614882990cc301e681e412. Latest consensus state: current_term: 1 leader_uuid: "db02c98c41614882990cc301e681e412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db02c98c41614882990cc301e681e412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43417 } } }
I20260927 15:12:15.056021 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:15.059657 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db02c98c41614882990cc301e681e412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db02c98c41614882990cc301e681e412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43417 } } }
I20260927 15:12:15.060086 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:15.060772 22204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:15.066747 22204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:15.067386 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:15.046440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48095 (local address 127.0.97.62:43417)
0927 15:12:15.046631 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:15.046636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:15.047146 (+ 510us) server_negotiation.cc:407] Connection header received
0927 15:12:15.047375 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:15.047379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:15.047448 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:15.047573 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:15.049009 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:15.049858 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:15.052285 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:15.052979 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:15.053132 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:15.056785 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:15.056796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:15.056803 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:15.056863 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:15.062744 (+ 5881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:15.063961 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:15.063971 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:15.063977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:15.064196 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:15.064802 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:15.064808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:15.064811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:15.065270 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:15.065580 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:15.066766 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0927 15:12:15.066963 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":81}
I20260927 15:12:15.080667 22204 catalog_manager.cc:1347] Generated new cluster ID: 0bf689a5e09243688d30f7854a318d73
I20260927 15:12:15.080965 22204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:15.097112 22204 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:15.098588 22204 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:15.107314 22204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db02c98c41614882990cc301e681e412: Generated new TSK 0
I20260927 15:12:15.108250 22204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:15.114691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22122
I20260927 15:12:15.115950 22184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:15.208426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22122
W20260927 15:12:15.226274 22107 connection.cc:535] client connection to 127.0.97.62:43417 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:43417): Connection reset by peer (error 104)
2026-09-27T15:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:15 dist-test-slave-g0w1 krb5kdc[22221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:15 dist-test-slave-g0w1 krb5kdc[22221](info): set up 2 sockets
Sep 27 15:12:15 dist-test-slave-g0w1 krb5kdc[22221](info): commencing operation
krb5kdc: starting...
W20260927 15:12:16.335812 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:16 dist-test-slave-g0w1 krb5kdc[22221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:16.609848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:16.959012 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:16.959452 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:16.959630 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:16.967281 22234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:16.967411 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:16.967458 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:16.967563 22234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:16.967631 22234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:16.978236 22234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:16.979751 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:16.981142 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:16.994063 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:16.994087 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:16.994372 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:16.995719 22234 server_base.cc:1034] running on GCE node
I20260927 15:12:16.996641 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:16.998286 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:16.999584 22234 hybrid_clock.cc:648] HybridClock initialized: now 1790521936999535 us; error 35 us; skew 500 ppm
I20260927 15:12:16.999977 22234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:16 dist-test-slave-g0w1 krb5kdc[22221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:17.003858 22234 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:17.005776 22234 webserver.cc:460] Webserver started at http://127.0.97.62:36009/ using document root <none> and password file <none>
I20260927 15:12:17.006584 22234 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:17.006747 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:17.007155 22234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:17.010494 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "d1d27c00aebe44b4aa9717d718d6559a"
format_stamp: "Formatted at 2026-09-27 15:12:17 on dist-test-slave-g0w1"
server_key: "a58941374cd574332ef13a88f238981f"
server_key_iv: "1457c562d669e236bf11bcf7f67c981c"
server_key_version: "encryptionkey@0"
I20260927 15:12:17.011456 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1d27c00aebe44b4aa9717d718d6559a"
format_stamp: "Formatted at 2026-09-27 15:12:17 on dist-test-slave-g0w1"
server_key: "a58941374cd574332ef13a88f238981f"
server_key_iv: "1457c562d669e236bf11bcf7f67c981c"
server_key_version: "encryptionkey@0"
I20260927 15:12:17.017535 22234 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 15:12:17.021992 22249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:17.023809 22234 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:12:17.024034 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d1d27c00aebe44b4aa9717d718d6559a"
format_stamp: "Formatted at 2026-09-27 15:12:17 on dist-test-slave-g0w1"
server_key: "a58941374cd574332ef13a88f238981f"
server_key_iv: "1457c562d669e236bf11bcf7f67c981c"
server_key_version: "encryptionkey@0"
I20260927 15:12:17.024276 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:17.057021 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:17.062089 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:17.062453 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:17.082526 22234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37687
I20260927 15:12:17.082551 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37687 every 8 connection(s)
I20260927 15:12:17.084342 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:17.087598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22234
I20260927 15:12:17.087790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:17.088331 388 external_mini_cluster.cc:1428] Setting key 8fa36b1d66ff5e1904db10a2d812b235
I20260927 15:12:17.089170 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:17.102860 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:17.090311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37647 (local address 127.0.97.62:37687)
0927 15:12:17.091216 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:17.091231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:17.091274 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:12:17.092884 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:17.092907 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:17.093410 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:17.093792 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:17.094468 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:17.096017 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:17.097944 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:17.098733 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:17.098883 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:17.100316 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:17.100325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:17.100341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:17.100969 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:17.101054 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:17.101080 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:17.101211 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:17.101598 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:12:17.101835 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":401,"threads_started":1}
I20260927 15:12:17.103801 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a: Bootstrap starting.
I20260927 15:12:17.106982 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:17.107174 22301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:17.108421 22301 log.cc:824] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:17.112125 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a: No bootstrap required, opened a new log
I20260927 15:12:17.119776 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:17.112341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56963 (local address 127.0.97.62:37687)
0927 15:12:17.112529 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:17.112535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:17.113035 (+ 500us) server_negotiation.cc:407] Connection header received
0927 15:12:17.113197 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:17.113202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:17.113280 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:17.113396 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:17.113859 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:17.115213 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:17.116640 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:17.117401 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:17.117555 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:17.118785 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:17.118795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:17.118800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:17.118829 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:17.118881 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:17.118903 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:17.119019 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:17.119331 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 15:12:17.119392 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:12:17.122433 22301 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d27c00aebe44b4aa9717d718d6559a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260927 15:12:17.122888 22301 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:17.122977 22301 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d27c00aebe44b4aa9717d718d6559a, State: Initialized, Role: FOLLOWER
I20260927 15:12:17.123613 22301 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d27c00aebe44b4aa9717d718d6559a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260927 15:12:17.123862 22301 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:17.124022 22301 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:17.124208 22301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:17.126734 22301 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d27c00aebe44b4aa9717d718d6559a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260927 15:12:17.127260 22301 leader_election.cc:302] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d27c00aebe44b4aa9717d718d6559a; no voters:
I20260927 15:12:17.127821 22301 leader_election.cc:288] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:17.128012 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:17.128850 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [term 1 LEADER]: Becoming Leader. State: Replica: d1d27c00aebe44b4aa9717d718d6559a, State: Running, Role: LEADER
I20260927 15:12:17.129540 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d27c00aebe44b4aa9717d718d6559a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260927 15:12:17.130174 22301 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:17.132915 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d27c00aebe44b4aa9717d718d6559a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d27c00aebe44b4aa9717d718d6559a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260927 15:12:17.133314 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:17.133771 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d27c00aebe44b4aa9717d718d6559a. Latest consensus state: current_term: 1 leader_uuid: "d1d27c00aebe44b4aa9717d718d6559a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d27c00aebe44b4aa9717d718d6559a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260927 15:12:17.134378 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:17.135795 22319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:17.141754 22319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:17.151499 22319 catalog_manager.cc:1347] Generated new cluster ID: 2957bd8d6bc7409f8a8793bb14bda3ce
I20260927 15:12:17.151643 22319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:17.186882 22319 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:17.189179 22319 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:17.196519 22319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d27c00aebe44b4aa9717d718d6559a: Generated new TSK 0
I20260927 15:12:17.197659 22319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:17.206664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22234
W20260927 15:12:17.208410 22296 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37647
W20260927 15:12:17.209429 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22234: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:17.209628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22234
W20260927 15:12:17.223718 22219 connection.cc:535] client connection to 127.0.97.62:37687 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37687): Connection reset by peer (error 104)
2026-09-27T15:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:17 dist-test-slave-g0w1 krb5kdc[22333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:17 dist-test-slave-g0w1 krb5kdc[22333](info): set up 2 sockets
Sep 27 15:12:17 dist-test-slave-g0w1 krb5kdc[22333](info): commencing operation
krb5kdc: starting...
W20260927 15:12:18.407660 388 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:18 dist-test-slave-g0w1 krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:18.678983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39039
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:19.019860 22346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:19.020201 22346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:19.020299 22346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:19.027716 22346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:19.027848 22346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:19.027894 22346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:19.027956 22346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:19.028002 22346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:19.038152 22346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:19.039501 22346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:19.040803 22346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:19.053574 22351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:19.053604 22352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:19.053562 22354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:19.055032 22346 server_base.cc:1034] running on GCE node
I20260927 15:12:19.055997 22346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:19.057611 22346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:19.058867 22346 hybrid_clock.cc:648] HybridClock initialized: now 1790521939058828 us; error 44 us; skew 500 ppm
I20260927 15:12:19.059321 22346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:19 dist-test-slave-g0w1 krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:19.074697 22346 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:19.076622 22346 webserver.cc:460] Webserver started at http://127.0.97.62:41603/ using document root <none> and password file <none>
I20260927 15:12:19.077407 22346 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:19.077565 22346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:19.077911 22346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:19.081664 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "0e2e147612c94090bff1febf520b15e6"
format_stamp: "Formatted at 2026-09-27 15:12:19 on dist-test-slave-g0w1"
server_key: "9fab361cded2f7bd68ee6d348bc05e6e"
server_key_iv: "759933c734f28b73605763fa5a807b8f"
server_key_version: "encryptionkey@0"
I20260927 15:12:19.082638 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e2e147612c94090bff1febf520b15e6"
format_stamp: "Formatted at 2026-09-27 15:12:19 on dist-test-slave-g0w1"
server_key: "9fab361cded2f7bd68ee6d348bc05e6e"
server_key_iv: "759933c734f28b73605763fa5a807b8f"
server_key_version: "encryptionkey@0"
I20260927 15:12:19.088385 22346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 15:12:19.092229 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:19.093744 22346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:12:19.093961 22346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "0e2e147612c94090bff1febf520b15e6"
format_stamp: "Formatted at 2026-09-27 15:12:19 on dist-test-slave-g0w1"
server_key: "9fab361cded2f7bd68ee6d348bc05e6e"
server_key_iv: "759933c734f28b73605763fa5a807b8f"
server_key_version: "encryptionkey@0"
I20260927 15:12:19.094197 22346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:19.100492 22346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:19.105465 22346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:19.105844 22346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:19.127012 22346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39039
I20260927 15:12:19.127044 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39039 every 8 connection(s)
I20260927 15:12:19.128754 22346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:19.132834 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:19.136255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22346
I20260927 15:12:19.136461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:19.136907 388 external_mini_cluster.cc:1428] Setting key b5811c36f4f8dd9742c4471ea1ea7444
I20260927 15:12:19.146539 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:19.138956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47511 (local address 127.0.97.62:39039)
0927 15:12:19.139906 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:19.139923 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:19.139992 (+ 69us) server_negotiation.cc:407] Connection header received
0927 15:12:19.142282 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:19.142320 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:19.142864 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:19.143404 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:19.144494 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:19.144502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:19.144527 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:19.144546 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:19.144617 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:19.144644 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:19.144896 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:19.145025 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:12:19.145226 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":341,"threads_started":1}
I20260927 15:12:19.147182 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6: Bootstrap starting.
I20260927 15:12:19.150494 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:19.150825 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:19.151922 22413 log.cc:824] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:19.155277 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6: No bootstrap required, opened a new log
I20260927 15:12:19.158174 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:19.155433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45013 (local address 127.0.97.62:39039)
0927 15:12:19.155627 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:19.155633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:19.155846 (+ 213us) server_negotiation.cc:407] Connection header received
0927 15:12:19.156056 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:19.156061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:19.156157 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:19.156272 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:19.157341 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:19.157346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:19.157351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:19.157372 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:19.157449 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:19.157470 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:19.157619 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:19.157801 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:12:19.157870 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:12:19.164079 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2e147612c94090bff1febf520b15e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260927 15:12:19.164678 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:19.164813 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2e147612c94090bff1febf520b15e6, State: Initialized, Role: FOLLOWER
I20260927 15:12:19.165512 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2e147612c94090bff1febf520b15e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260927 15:12:19.165812 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:19.165966 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:19.166147 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:19.168794 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2e147612c94090bff1febf520b15e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260927 15:12:19.169348 22413 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2e147612c94090bff1febf520b15e6; no voters:
I20260927 15:12:19.169971 22413 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:19.170135 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:19.170965 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0e2e147612c94090bff1febf520b15e6, State: Running, Role: LEADER
I20260927 15:12:19.171700 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2e147612c94090bff1febf520b15e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260927 15:12:19.172369 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:19.174729 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e2e147612c94090bff1febf520b15e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2e147612c94090bff1febf520b15e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } } }
I20260927 15:12:19.175208 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:19.174716 22425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2e147612c94090bff1febf520b15e6. Latest consensus state: current_term: 1 leader_uuid: "0e2e147612c94090bff1febf520b15e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2e147612c94090bff1febf520b15e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } } }
I20260927 15:12:19.176915 22425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:19.179495 22431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:19.185489 22431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:19.194010 22431 catalog_manager.cc:1347] Generated new cluster ID: d893f4fa149946cc9c1848a2e8cfe39f
I20260927 15:12:19.194136 22431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:19.209354 22431 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:19.209834 22431 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:19.219604 22431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2e147612c94090bff1febf520b15e6: Generated new TSK 0
I20260927 15:12:19.220597 22431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:19.237339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22346
W20260927 15:12:19.238551 22408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47511
W20260927 15:12:19.239346 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:19.239506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22346
2026-09-27T15:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:19 dist-test-slave-g0w1 krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:19 dist-test-slave-g0w1 krb5kdc[22444](info): set up 2 sockets
Sep 27 15:12:19 dist-test-slave-g0w1 krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260927 15:12:20.443642 388 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:20 dist-test-slave-g0w1 krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:20.719285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:21.062949 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:21.063318 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:21.063491 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:21.071033 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:21.071224 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:21.071323 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:21.071441 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:21.071504 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:21.081600 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:21.083001 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:21.084427 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:21.096719 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:21.096666 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:21.096989 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:21.097797 22457 server_base.cc:1034] running on GCE node
I20260927 15:12:21.098627 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:21.100242 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:21.101498 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790521941101458 us; error 40 us; skew 500 ppm
I20260927 15:12:21.101874 22457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:21 dist-test-slave-g0w1 krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:21.105216 22457 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:21.106860 22457 webserver.cc:460] Webserver started at http://127.0.97.62:34115/ using document root <none> and password file <none>
I20260927 15:12:21.107646 22457 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:21.107791 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:21.108115 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:21.111285 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "b15d495fab18470e83327152aef57c45"
format_stamp: "Formatted at 2026-09-27 15:12:21 on dist-test-slave-g0w1"
server_key: "90cd0249dfadd1a4fc06ab637f57108d"
server_key_iv: "bf13e9703d9c1a432ff65bca0d355c24"
server_key_version: "encryptionkey@0"
I20260927 15:12:21.112210 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "b15d495fab18470e83327152aef57c45"
format_stamp: "Formatted at 2026-09-27 15:12:21 on dist-test-slave-g0w1"
server_key: "90cd0249dfadd1a4fc06ab637f57108d"
server_key_iv: "bf13e9703d9c1a432ff65bca0d355c24"
server_key_version: "encryptionkey@0"
I20260927 15:12:21.117789 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:12:21.121681 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:21.123427 22457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:12:21.123661 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "b15d495fab18470e83327152aef57c45"
format_stamp: "Formatted at 2026-09-27 15:12:21 on dist-test-slave-g0w1"
server_key: "90cd0249dfadd1a4fc06ab637f57108d"
server_key_iv: "bf13e9703d9c1a432ff65bca0d355c24"
server_key_version: "encryptionkey@0"
I20260927 15:12:21.123904 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:21.141021 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:21.145780 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:21.146127 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:21.166029 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42509
I20260927 15:12:21.166061 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42509 every 8 connection(s)
I20260927 15:12:21.167575 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:21.171397 22524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:21.175959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22457
I20260927 15:12:21.176147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:21.176695 388 external_mini_cluster.cc:1428] Setting key bae72863f587fb8ed62c8149557d3aa7
I20260927 15:12:21.184916 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45: Bootstrap starting.
I20260927 15:12:21.188619 22524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:21.189767 22524 log.cc:824] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:12:21 dist-test-slave-g0w1 krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:21.193255 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45: No bootstrap required, opened a new log
I20260927 15:12:21.201752 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:21.178599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44175 (local address 127.0.97.62:42509)
0927 15:12:21.179477 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:21.179490 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:21.179538 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:12:21.181178 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:21.181204 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:21.181674 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:21.182145 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:21.183792 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:21.185222 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:21.186732 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:21.187596 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:21.187809 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:21.192173 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:21.192193 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:21.192218 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:21.192278 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:21.196715 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:21.197560 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:21.197571 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:21.197582 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:21.197812 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:21.198187 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:21.198192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:21.198196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:21.198951 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:21.199300 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:21.199728 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 15:12:21.200260 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":439,"threads_started":1}
I20260927 15:12:21.204520 22524 raft_consensus.cc:348] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d495fab18470e83327152aef57c45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42509 } }
I20260927 15:12:21.205128 22524 raft_consensus.cc:374] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:21.205255 22524 raft_consensus.cc:729] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15d495fab18470e83327152aef57c45, State: Initialized, Role: FOLLOWER
I20260927 15:12:21.206004 22524 consensus_queue.cc:260] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d495fab18470e83327152aef57c45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42509 } }
I20260927 15:12:21.206399 22524 raft_consensus.cc:388] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:21.206547 22524 raft_consensus.cc:482] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:21.206732 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:21.206720 22524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:21.209249 22524 raft_consensus.cc:504] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d495fab18470e83327152aef57c45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42509 } }
I20260927 15:12:21.209952 22524 leader_election.cc:302] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b15d495fab18470e83327152aef57c45; no voters:
I20260927 15:12:21.210661 22524 leader_election.cc:288] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:21.210780 22533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:21.212857 22533 raft_consensus.cc:686] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [term 1 LEADER]: Becoming Leader. State: Replica: b15d495fab18470e83327152aef57c45, State: Running, Role: LEADER
I20260927 15:12:21.213737 22533 consensus_queue.cc:237] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d495fab18470e83327152aef57c45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42509 } }
I20260927 15:12:21.213965 22524 sys_catalog.cc:564] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:21.216487 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b15d495fab18470e83327152aef57c45. Latest consensus state: current_term: 1 leader_uuid: "b15d495fab18470e83327152aef57c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d495fab18470e83327152aef57c45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42509 } } }
I20260927 15:12:21.217475 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:21.217136 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b15d495fab18470e83327152aef57c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d495fab18470e83327152aef57c45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42509 } } }
I20260927 15:12:21.218025 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:21.221038 22539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:21.231302 22539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:21.234031 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:21.212642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46315 (local address 127.0.97.62:42509)
0927 15:12:21.212882 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:21.212890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:21.212915 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:12:21.212984 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:21.212989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:21.213079 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:21.213176 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:21.215619 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:21.217266 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:21.219884 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:21.220929 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:21.221082 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:21.225574 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:21.225588 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:21.225596 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:21.225657 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:21.229860 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:21.230861 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:21.230865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:21.230868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:21.231048 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:21.231557 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:21.231562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:21.231566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:21.232012 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:21.233227 (+ 1215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:21.233379 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:12:21.233627 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:12:21.242970 22539 catalog_manager.cc:1347] Generated new cluster ID: 9e7b3bab3131480ca964a4476f34b8af
I20260927 15:12:21.243151 22539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:21.253886 22539 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:21.255838 22539 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:21.265214 22539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b15d495fab18470e83327152aef57c45: Generated new TSK 0
I20260927 15:12:21.266356 22539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:21.274577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22457
I20260927 15:12:21.275925 22519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:21.368397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22457
2026-09-27T15:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2129 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:21 dist-test-slave-g0w1 krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:21 dist-test-slave-g0w1 krb5kdc[22556](info): set up 2 sockets
Sep 27 15:12:21 dist-test-slave-g0w1 krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260927 15:12:22.476162 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:22 dist-test-slave-g0w1 krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:22.750597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:23.094017 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:23.094323 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:23.094506 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:23.101949 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:23.102092 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:23.102138 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:23.102254 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:23.102322 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:23.112814 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:23.114217 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:23.115599 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:23.128319 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:23.128338 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:23.128259 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:23.129349 22569 server_base.cc:1034] running on GCE node
I20260927 15:12:23.130235 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:23.131913 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:23.133129 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790521943133087 us; error 43 us; skew 500 ppm
I20260927 15:12:23.133544 22569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:23 dist-test-slave-g0w1 krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:23.136963 22569 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:23.138711 22569 webserver.cc:460] Webserver started at http://127.0.97.62:35319/ using document root <none> and password file <none>
I20260927 15:12:23.139572 22569 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:23.139732 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:23.140055 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:23.143421 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "839abb35eed5499996e55a707fe92f11"
format_stamp: "Formatted at 2026-09-27 15:12:23 on dist-test-slave-g0w1"
server_key: "448867478e768f974ba52993ab8a4797"
server_key_iv: "55dcb2c4539734275b990a3c9b81c917"
server_key_version: "encryptionkey@0"
I20260927 15:12:23.144276 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "839abb35eed5499996e55a707fe92f11"
format_stamp: "Formatted at 2026-09-27 15:12:23 on dist-test-slave-g0w1"
server_key: "448867478e768f974ba52993ab8a4797"
server_key_iv: "55dcb2c4539734275b990a3c9b81c917"
server_key_version: "encryptionkey@0"
I20260927 15:12:23.149837 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:12:23.153621 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:23.155124 22569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 15:12:23.155360 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "839abb35eed5499996e55a707fe92f11"
format_stamp: "Formatted at 2026-09-27 15:12:23 on dist-test-slave-g0w1"
server_key: "448867478e768f974ba52993ab8a4797"
server_key_iv: "55dcb2c4539734275b990a3c9b81c917"
server_key_version: "encryptionkey@0"
I20260927 15:12:23.155592 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:23.181669 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:23.186141 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:23.186482 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:23.206462 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40989
I20260927 15:12:23.206486 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40989 every 8 connection(s)
I20260927 15:12:23.208055 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:23.212160 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:23.217216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22569
I20260927 15:12:23.217394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:23.217813 388 external_mini_cluster.cc:1428] Setting key 6ea24d6da45ca5bd618f03b981a06dbd
I20260927 15:12:23.227239 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11: Bootstrap starting.
I20260927 15:12:23.231395 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:23.232633 22636 log.cc:824] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:23.232769 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:23.219833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36811 (local address 127.0.97.62:40989)
0927 15:12:23.220670 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:23.220689 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:23.220743 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:12:23.222855 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:23.222888 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:23.223453 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:23.223850 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:23.224480 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:23.225857 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:23.227854 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:23.228582 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:23.228730 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:23.230138 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:23.230146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:23.230162 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:23.230743 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:23.230837 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:23.230866 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:23.230969 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:23.231277 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 15:12:23.231611 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":352,"threads_started":1}
I20260927 15:12:23.235895 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11: No bootstrap required, opened a new log
I20260927 15:12:23.238385 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:23.245993 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839abb35eed5499996e55a707fe92f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260927 15:12:23.246469 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:23.246609 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839abb35eed5499996e55a707fe92f11, State: Initialized, Role: FOLLOWER
I20260927 15:12:23.247416 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839abb35eed5499996e55a707fe92f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260927 15:12:23.247731 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:23.247903 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:23.248101 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:23.249684 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:23.242878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39699 (local address 127.0.97.62:40989)
0927 15:12:23.243165 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:23.243178 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:23.243469 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:12:23.243723 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:23.243727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:23.243797 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:23.243914 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:23.244474 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:23.245208 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:23.246565 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:23.247272 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:23.247415 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:23.248558 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:23.248565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:23.248571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:23.248594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:23.248643 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:23.248663 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:23.248771 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:23.249217 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 15:12:23.249330 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 15:12:23.251219 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839abb35eed5499996e55a707fe92f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260927 15:12:23.251794 22636 leader_election.cc:302] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839abb35eed5499996e55a707fe92f11; no voters:
I20260927 15:12:23.252331 22636 leader_election.cc:288] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:23.252519 22646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:23.254490 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:23.259836 22646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [term 1 LEADER]: Becoming Leader. State: Replica: 839abb35eed5499996e55a707fe92f11, State: Running, Role: LEADER
I20260927 15:12:23.260506 22646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839abb35eed5499996e55a707fe92f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260927 15:12:23.263689 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "839abb35eed5499996e55a707fe92f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839abb35eed5499996e55a707fe92f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } } }
I20260927 15:12:23.264103 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:23.264853 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839abb35eed5499996e55a707fe92f11. Latest consensus state: current_term: 1 leader_uuid: "839abb35eed5499996e55a707fe92f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839abb35eed5499996e55a707fe92f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } } }
I20260927 15:12:23.265297 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11 [sys.catalog]: This master's current role is: LEADER
W20260927 15:12:23.269721 22660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:12:23.269871 22660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:12:23.270069 22661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:23.272424 22661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:23.282042 22661 catalog_manager.cc:1347] Generated new cluster ID: 0a662cb1ffc04cb4b6b4dbc28b9a234c
I20260927 15:12:23.282168 22661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:23.320142 22661 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:23.321586 22661 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:23.336493 22661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839abb35eed5499996e55a707fe92f11: Generated new TSK 0
I20260927 15:12:23.337623 22661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:23.356308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22569
W20260927 15:12:23.357514 22631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36811
W20260927 15:12:23.358314 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:23.358482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22569
2026-09-27T15:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:23 dist-test-slave-g0w1 krb5kdc[22667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:23 dist-test-slave-g0w1 krb5kdc[22667](info): set up 2 sockets
Sep 27 15:12:23 dist-test-slave-g0w1 krb5kdc[22667](info): commencing operation
krb5kdc: starting...
W20260927 15:12:24.479696 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:24 dist-test-slave-g0w1 krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:24.748600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44273
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:25.100360 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:25.100701 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:25.100798 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:25.108645 22680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:25.108786 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:25.108834 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:25.108889 22680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:25.108935 22680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:25.119572 22680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:25.120913 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:25.122288 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:25.135511 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:25.135558 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:25.135507 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:25.136889 22680 server_base.cc:1034] running on GCE node
I20260927 15:12:25.137768 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:25.139519 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:25.140774 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790521945140716 us; error 44 us; skew 500 ppm
I20260927 15:12:25.141198 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:25 dist-test-slave-g0w1 krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:25.149232 22680 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:25.151156 22680 webserver.cc:460] Webserver started at http://127.0.97.62:37489/ using document root <none> and password file <none>
I20260927 15:12:25.151957 22680 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:25.152122 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:25.152494 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:25.155831 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "bead50f472f649bbba4add754c710c9d"
format_stamp: "Formatted at 2026-09-27 15:12:25 on dist-test-slave-g0w1"
server_key: "b4c84d551a89d933ccebccc4c4db07fe"
server_key_iv: "4c98176a072ee30c15df5a9016d5c927"
server_key_version: "encryptionkey@0"
I20260927 15:12:25.156677 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "bead50f472f649bbba4add754c710c9d"
format_stamp: "Formatted at 2026-09-27 15:12:25 on dist-test-slave-g0w1"
server_key: "b4c84d551a89d933ccebccc4c4db07fe"
server_key_iv: "4c98176a072ee30c15df5a9016d5c927"
server_key_version: "encryptionkey@0"
I20260927 15:12:25.162519 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:12:25.166729 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:25.168386 22680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:12:25.168632 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "bead50f472f649bbba4add754c710c9d"
format_stamp: "Formatted at 2026-09-27 15:12:25 on dist-test-slave-g0w1"
server_key: "b4c84d551a89d933ccebccc4c4db07fe"
server_key_iv: "4c98176a072ee30c15df5a9016d5c927"
server_key_version: "encryptionkey@0"
I20260927 15:12:25.168869 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:25.175698 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:25.180712 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:25.181064 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:25.201175 22680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44273
I20260927 15:12:25.201208 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44273 every 8 connection(s)
I20260927 15:12:25.202759 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:25.206280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22680
I20260927 15:12:25.206460 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:25.206996 388 external_mini_cluster.cc:1428] Setting key 9ee2677f30a3f319e6c1e6eeeef12dd4
I20260927 15:12:25.207268 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:25.216161 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:25.208975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52331 (local address 127.0.97.62:44273)
0927 15:12:25.209827 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:25.209841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:25.209884 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:12:25.212266 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:25.212292 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:25.212719 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:25.213213 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:25.214361 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:25.214368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:25.214382 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:25.214398 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:25.214457 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:25.214481 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:25.214750 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:25.214910 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 15:12:25.215166 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":358,"threads_started":1}
I20260927 15:12:25.220407 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:25.221946 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d: Bootstrap starting.
I20260927 15:12:25.226133 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:25.227434 22747 log.cc:824] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:25.228701 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:25.225853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39917 (local address 127.0.97.62:44273)
0927 15:12:25.226056 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:25.226065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:25.226543 (+ 478us) server_negotiation.cc:407] Connection header received
0927 15:12:25.226759 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:25.226764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:25.226838 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:25.226955 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:25.228025 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:25.228031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:25.228036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:12:25.228055 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:25.228106 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:12:25.228121 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:25.228246 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:25.228352 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 15:12:25.228415 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:12:25.231309 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d: No bootstrap required, opened a new log
I20260927 15:12:25.239288 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead50f472f649bbba4add754c710c9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44273 } }
I20260927 15:12:25.239857 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:25.239984 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bead50f472f649bbba4add754c710c9d, State: Initialized, Role: FOLLOWER
I20260927 15:12:25.240728 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead50f472f649bbba4add754c710c9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44273 } }
I20260927 15:12:25.240973 22747 raft_consensus.cc:388] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:25.241151 22747 raft_consensus.cc:482] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:25.241356 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:25.243850 22747 raft_consensus.cc:504] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead50f472f649bbba4add754c710c9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44273 } }
I20260927 15:12:25.244441 22747 leader_election.cc:302] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bead50f472f649bbba4add754c710c9d; no voters:
I20260927 15:12:25.245078 22747 leader_election.cc:288] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:25.245263 22757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:25.246613 22757 raft_consensus.cc:686] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [term 1 LEADER]: Becoming Leader. State: Replica: bead50f472f649bbba4add754c710c9d, State: Running, Role: LEADER
I20260927 15:12:25.247363 22757 consensus_queue.cc:237] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead50f472f649bbba4add754c710c9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44273 } }
I20260927 15:12:25.247584 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:25.250830 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bead50f472f649bbba4add754c710c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead50f472f649bbba4add754c710c9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44273 } } }
I20260927 15:12:25.251436 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:25.250828 22759 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bead50f472f649bbba4add754c710c9d. Latest consensus state: current_term: 1 leader_uuid: "bead50f472f649bbba4add754c710c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead50f472f649bbba4add754c710c9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44273 } } }
I20260927 15:12:25.253101 22759 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:25.256402 22767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:25.261582 22767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:25.270531 22767 catalog_manager.cc:1347] Generated new cluster ID: 91fd52ebce3f4015b18717fad9feb3ba
I20260927 15:12:25.270689 22767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:25.288913 22767 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:25.289489 22767 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:25.304489 22767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bead50f472f649bbba4add754c710c9d: Generated new TSK 0
I20260927 15:12:25.305703 22767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:25.315011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22680
W20260927 15:12:25.316507 22742 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52331
W20260927 15:12:25.317322 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:12:25.317494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22680
2026-09-27T15:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1958 ms)
[----------] 18 tests from AuthTokenIssuingTest (36472 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:25 dist-test-slave-g0w1 krb5kdc[22778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:25 dist-test-slave-g0w1 krb5kdc[22778](info): set up 2 sockets
Sep 27 15:12:25 dist-test-slave-g0w1 krb5kdc[22778](info): commencing operation
krb5kdc: starting...
W20260927 15:12:26.547832 388 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:26 dist-test-slave-g0w1 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:26.822885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33761
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33761,127.0.97.61:37019,127.0.97.60:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:27.179822 22791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:12:27.180181 22791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:27.180338 22791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:27.180482 22791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:27.188155 22791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:27.188266 22791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:27.188387 22791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:27.188475 22791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:27.188570 22791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:27.199117 22791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33761,127.0.97.61:37019,127.0.97.60:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:27.200582 22791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:27.202183 22791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:27.215163 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:27.215242 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:27.215127 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:27.216048 22791 server_base.cc:1034] running on GCE node
I20260927 15:12:27.216885 22791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:27.218518 22791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:27.219776 22791 hybrid_clock.cc:648] HybridClock initialized: now 1790521947219678 us; error 66 us; skew 500 ppm
I20260927 15:12:27.220220 22791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:27 dist-test-slave-g0w1 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:27.223877 22791 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:27.225683 22791 webserver.cc:460] Webserver started at http://127.0.97.62:42569/ using document root <none> and password file <none>
I20260927 15:12:27.226480 22791 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:27.226573 22791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:27.226914 22791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:27.231127 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d"
format_stamp: "Formatted at 2026-09-27 15:12:27 on dist-test-slave-g0w1"
server_key: "bf009386a584a37e404f103bdb7b4a6d"
server_key_iv: "016eb18beef9749be52f2466ad9985a0"
server_key_version: "encryptionkey@0"
I20260927 15:12:27.232187 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d"
format_stamp: "Formatted at 2026-09-27 15:12:27 on dist-test-slave-g0w1"
server_key: "bf009386a584a37e404f103bdb7b4a6d"
server_key_iv: "016eb18beef9749be52f2466ad9985a0"
server_key_version: "encryptionkey@0"
I20260927 15:12:27.238046 22791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:12:27.242027 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:27.243682 22791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:27.243896 22791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d"
format_stamp: "Formatted at 2026-09-27 15:12:27 on dist-test-slave-g0w1"
server_key: "bf009386a584a37e404f103bdb7b4a6d"
server_key_iv: "016eb18beef9749be52f2466ad9985a0"
server_key_version: "encryptionkey@0"
I20260927 15:12:27.244139 22791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:27.267025 22791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:27.271798 22791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:27.272114 22791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:27.292376 22791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33761
I20260927 15:12:27.292394 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33761 every 8 connection(s)
I20260927 15:12:27.293906 22791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:27.297950 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:27.300717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22791
I20260927 15:12:27.300947 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:27.301751 388 external_mini_cluster.cc:1428] Setting key 952ab9ac8fae89546a653a11f1516047
I20260927 15:12:27.307981 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:27 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 15:12:27.338418 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.312988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.62:33761)
0927 15:12:27.315954 (+ 2966us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:27.315964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:27.316030 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:12:27.317447 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:27.317458 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:27.317904 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:27.318155 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:27.319847 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.321898 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.324048 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.325112 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.325340 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:27.329333 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:27.329348 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:27.329370 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:27.329424 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:27.333708 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:27.334590 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:27.334603 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:27.334606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:27.334842 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:27.335315 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:27.335319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:27.335321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:27.335999 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:27.336169 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:27.336840 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 15:12:27.337314 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":2713,"thread_start_us":2530,"threads_started":1}
I20260927 15:12:27.338487 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.312383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33761 (local address 127.0.0.1:48280)
0927 15:12:27.313555 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.313628 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:27.316865 (+ 3237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:27.318179 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:27.318186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:27.319007 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:27.319545 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:27.319580 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.322181 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.322187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:27.323877 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:27.323893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.325367 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.325373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:27.325505 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:27.326286 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:27.326314 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:27.329022 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:27.333912 (+ 4890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:27.333925 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:27.333936 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:27.334408 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:27.334965 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:27.334969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:27.334971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:27.335199 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:27.336136 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:27.336143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:27.336394 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 15:12:27.336616 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 15:12:27.337129 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"spinlock_wait_cycles":17280,"thread_start_us":448,"threads_started":1}
I20260927 15:12:27.341727 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:27.342990 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.342513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39270)
0927 15:12:27.342721 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.342848 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 15:12:27.343447 22807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
I20260927 15:12:27.346264 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.345009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39272)
0927 15:12:27.345994 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.346101 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853}
W20260927 15:12:27.346599 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37019: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 15:12:27.372133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37019
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33761,127.0.97.61:37019,127.0.97.60:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:12:27.396029 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } attempt: 1
I20260927 15:12:27.397388 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.396770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39282)
0927 15:12:27.397050 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.397166 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 15:12:27.398629 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.398177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39284)
0927 15:12:27.398369 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.398463 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 15:12:27.398941 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37019: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
I20260927 15:12:27.467263 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } attempt: 2
I20260927 15:12:27.468621 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.468031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39290)
0927 15:12:27.468331 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.468445 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:12:27.469832 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.469389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39296)
0927 15:12:27.469567 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.469671 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:12:27.470077 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37019: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
I20260927 15:12:27.561396 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } attempt: 3
I20260927 15:12:27.562780 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.562165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39298)
0927 15:12:27.562438 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.562565 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 15:12:27.564002 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.563527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39304)
0927 15:12:27.563692 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.563793 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:12:27.564280 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37019: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
I20260927 15:12:27.707587 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } attempt: 4
I20260927 15:12:27.709036 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.708367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39310)
0927 15:12:27.708745 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.708859 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 15:12:27.710417 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.709938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39312)
0927 15:12:27.710125 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.710236 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 15:12:27.710662 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37019: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37019: connect: Connection refused (error 111)
W20260927 15:12:27.729455 22865 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:12:27.729839 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:27.729951 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:27.730122 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:27.737896 22865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:27.738013 22865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:27.738097 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:27.738171 22865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:27.738267 22865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:27.749475 22865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33761,127.0.97.61:37019,127.0.97.60:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:27.750893 22865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:27.752440 22865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:27.766036 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:27.766036 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:27.766058 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:27.767298 22865 server_base.cc:1034] running on GCE node
I20260927 15:12:27.768210 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:27.769837 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:27.771160 22865 hybrid_clock.cc:648] HybridClock initialized: now 1790521947771010 us; error 76 us; skew 500 ppm
I20260927 15:12:27.771553 22865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:27 dist-test-slave-g0w1 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:27.775002 22865 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 15:12:27.776572 22865 webserver.cc:460] Webserver started at http://127.0.97.61:44951/ using document root <none> and password file <none>
I20260927 15:12:27.777383 22865 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:27.777527 22865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:27.777876 22865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:27.781227 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data/instance:
uuid: "6ea3ee92abe5412ebc891c11347c5955"
format_stamp: "Formatted at 2026-09-27 15:12:27 on dist-test-slave-g0w1"
server_key: "1a285296918076d24c0954e2c3938967"
server_key_iv: "cd67f76e51b584bb8dafb4091f7532ca"
server_key_version: "encryptionkey@0"
I20260927 15:12:27.782146 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal/instance:
uuid: "6ea3ee92abe5412ebc891c11347c5955"
format_stamp: "Formatted at 2026-09-27 15:12:27 on dist-test-slave-g0w1"
server_key: "1a285296918076d24c0954e2c3938967"
server_key_iv: "cd67f76e51b584bb8dafb4091f7532ca"
server_key_version: "encryptionkey@0"
I20260927 15:12:27.788002 22865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:12:27.791958 22880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:27.793493 22865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:12:27.793710 22865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal
uuid: "6ea3ee92abe5412ebc891c11347c5955"
format_stamp: "Formatted at 2026-09-27 15:12:27 on dist-test-slave-g0w1"
server_key: "1a285296918076d24c0954e2c3938967"
server_key_iv: "cd67f76e51b584bb8dafb4091f7532ca"
server_key_version: "encryptionkey@0"
I20260927 15:12:27.793960 22865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:27.810350 22865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:27.815277 22865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:27.815608 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:27.837082 22865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37019
I20260927 15:12:27.837100 22931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37019 every 8 connection(s)
I20260927 15:12:27.838647 22865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
I20260927 15:12:27.840123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22865
I20260927 15:12:27.840324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-1/wal/instance
I20260927 15:12:27.840862 388 external_mini_cluster.cc:1428] Setting key 300278bcbbaa5cf866237ec8e9b9a34d
I20260927 15:12:27.843979 22932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:27.854064 22932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 15:12:27 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:27.881080 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.857137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.97.62:33761)
0927 15:12:27.857729 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:27.857733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:27.858919 (+ 1186us) server_negotiation.cc:407] Connection header received
0927 15:12:27.861146 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:27.861153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:27.861256 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:27.861387 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:27.863233 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.863993 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.866903 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.867420 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.867533 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:27.873932 (+ 6399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:27.873938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:27.873942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:27.873996 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:27.876982 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:27.878226 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:27.878231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:27.878234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:27.878381 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:27.878952 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:27.878956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:27.878958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:27.879367 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:27.879491 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:27.880509 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0927 15:12:27.880697 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":351,"threads_started":1}
I20260927 15:12:27.882217 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.857828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33761 (local address 127.0.0.1:48288)
0927 15:12:27.858718 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.858769 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:27.860797 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:27.861674 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:27.861684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:27.862277 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:27.862948 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:27.862969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.864234 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.864241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:27.866658 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:27.866683 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.867690 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.867698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:27.868114 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:27.868929 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:27.868965 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:27.873574 (+ 4609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:27.877214 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:27.877236 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:27.877260 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:27.878052 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:27.878518 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:27.878526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:27.878532 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:27.878823 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:27.879519 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:27.879532 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:27.880063 (+ 531us) client_negotiation.cc:769] Sending connection context
0927 15:12:27.880413 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 15:12:27.881026 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":371,"threads_started":1}
I20260927 15:12:27.884114 22932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:27 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:27.900192 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.884955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39314)
0927 15:12:27.885167 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.885186 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:27.885294 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:27.886427 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:27.886432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:27.886934 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:27.887159 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:27.887170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.888647 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.888655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:27.889522 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:27.889533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.890443 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.890449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:27.890692 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:27.891432 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:27.891450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:27.893993 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:27.897548 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:27.897553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:27.897557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:27.897913 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:27.898385 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:27.898392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:27.898396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:27.898562 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:27.899333 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:27.899345 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:27.899524 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 15:12:27.899649 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:12:27.899885 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":46}
I20260927 15:12:27.900492 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.885237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.0.97.61:37019)
0927 15:12:27.885936 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:27.885943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:27.885971 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:12:27.886020 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:27.886025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:27.886253 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:27.886448 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:27.887345 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.888482 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.889670 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:27.890293 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:27.890432 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:27.894242 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:27.894254 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:27.894272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:27.894299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:27.897391 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:27.898082 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:27.898091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:27.898101 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:27.898279 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:27.898736 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:27.898744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:27.898747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:27.899198 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:27.899426 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:27.899794 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 15:12:27.900088 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":544,"thread_start_us":388,"threads_started":1}
I20260927 15:12:27.903039 22932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:27.904402 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.903886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57436)
0927 15:12:27.904057 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.904179 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:12:27.904697 22883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 15:12:27.906142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38595
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33761,127.0.97.61:37019,127.0.97.60:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:12:27.906584 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.906129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57450)
0927 15:12:27.906323 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.906433 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:12:27.906908 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:27.956246 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 1
I20260927 15:12:27.957809 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.957078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57454)
0927 15:12:27.957432 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.957576 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 15:12:27.959754 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:27.958851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57466)
0927 15:12:27.959330 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:27.959494 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248}
W20260927 15:12:27.960055 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:28.010032 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } attempt: 5
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:28.025117 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.010885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39320)
0927 15:12:28.011166 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.011191 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:28.011338 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:28.011768 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:28.011776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:28.012263 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:28.012497 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.012508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.013671 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.013677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.014565 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.014577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.015584 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.015591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.015804 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.016490 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:28.016508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:28.018785 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:28.022145 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.022154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.022161 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.022843 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.023381 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.023386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.023390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.023566 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.024108 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:28.024113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:28.024283 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 15:12:28.024448 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:12:28.024779 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:12:28.025142 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.011005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.97.61:37019)
0927 15:12:28.011242 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.011248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.011271 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:12:28.011486 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.011498 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.011606 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.011726 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.012682 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.013482 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.014821 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.015414 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.015563 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.019026 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.019030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.019034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.019118 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.021959 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.023001 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.023006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.023008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.023219 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.023684 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.023687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.023689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.023975 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.024084 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.024604 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.024883 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 15:12:28.026319 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:28.027568 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.027062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57482)
0927 15:12:28.027293 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.027392 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 15:12:28.028311 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 2
I20260927 15:12:28.028734 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.028318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57498)
0927 15:12:28.028476 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.028572 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:12:28.028940 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:28.029393 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.028836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57502)
0927 15:12:28.029183 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.029260 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260927 15:12:28.030357 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.029997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57518)
0927 15:12:28.030151 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.030234 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 15:12:28.030565 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:28.080210 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 1
I20260927 15:12:28.081396 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.080917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57520)
0927 15:12:28.081125 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.081231 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 15:12:28.082485 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.082139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57522)
0927 15:12:28.082303 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.082372 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:12:28.082703 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:28.121869 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 3
I20260927 15:12:28.123199 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.122630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57524)
0927 15:12:28.122851 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.122955 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 15:12:28.124708 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.124012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57540)
0927 15:12:28.124358 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.124563 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
W20260927 15:12:28.125015 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:28.150943 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 2
I20260927 15:12:28.152199 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.151706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57554)
0927 15:12:28.151952 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.152046 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:12:28.153262 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.152915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57570)
0927 15:12:28.153071 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.153148 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:12:28.153481 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
W20260927 15:12:28.258908 22940 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:12:28.259392 22940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:28.259524 22940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:28.259691 22940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:12:28.259671 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 3
I20260927 15:12:28.260756 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.260300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57580)
0927 15:12:28.260495 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.260606 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:12:28.261770 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.261390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57588)
0927 15:12:28.261597 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.261669 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 15:12:28.261972 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
W20260927 15:12:28.267719 22940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:28.267849 22940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:28.267970 22940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:28.268054 22940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:28.268162 22940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:28.268296 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 4
I20260927 15:12:28.269332 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.268883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57602)
0927 15:12:28.269100 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.269179 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:12:28.270221 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.269921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57616)
0927 15:12:28.270055 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.270125 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:12:28.270432 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38595: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38595: connect: Connection refused (error 111)
I20260927 15:12:28.278843 22940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33761,127.0.97.61:37019,127.0.97.60:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:28.280247 22940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:28.281757 22940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:28.293957 22946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:28.294221 22945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:28.294902 22948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:28.295166 22940 server_base.cc:1034] running on GCE node
I20260927 15:12:28.296015 22940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:28.297473 22940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:28.298817 22940 hybrid_clock.cc:648] HybridClock initialized: now 1790521948298716 us; error 55 us; skew 500 ppm
I20260927 15:12:28.299261 22940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:28.302453 22940 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 15:12:28.304033 22940 webserver.cc:460] Webserver started at http://127.0.97.60:45225/ using document root <none> and password file <none>
I20260927 15:12:28.304693 22940 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:28.304778 22940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:28.305047 22940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:28.308063 22940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data/instance:
uuid: "80408fd1e20c4cfdbe52c637cfd2e014"
format_stamp: "Formatted at 2026-09-27 15:12:28 on dist-test-slave-g0w1"
server_key: "3998c5f5ca3be7201811c716f94f4f14"
server_key_iv: "d466368e90c895b4dd91188337436777"
server_key_version: "encryptionkey@0"
I20260927 15:12:28.308811 22940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal/instance:
uuid: "80408fd1e20c4cfdbe52c637cfd2e014"
format_stamp: "Formatted at 2026-09-27 15:12:28 on dist-test-slave-g0w1"
server_key: "3998c5f5ca3be7201811c716f94f4f14"
server_key_iv: "d466368e90c895b4dd91188337436777"
server_key_version: "encryptionkey@0"
I20260927 15:12:28.314030 22940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:12:28.317837 22955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:28.319670 22940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:12:28.319829 22940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal
uuid: "80408fd1e20c4cfdbe52c637cfd2e014"
format_stamp: "Formatted at 2026-09-27 15:12:28 on dist-test-slave-g0w1"
server_key: "3998c5f5ca3be7201811c716f94f4f14"
server_key_iv: "d466368e90c895b4dd91188337436777"
server_key_version: "encryptionkey@0"
I20260927 15:12:28.320026 22940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:28.333599 22940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:28.338147 22940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:28.338378 22940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:28.358004 22940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38595
I20260927 15:12:28.358024 23006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38595 every 8 connection(s)
I20260927 15:12:28.359483 22940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
I20260927 15:12:28.362697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 22940
I20260927 15:12:28.362890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521577514430-388-0/minicluster-data/master-2/wal/instance
I20260927 15:12:28.363454 388 external_mini_cluster.cc:1428] Setting key 13b2efdfe011cd0a323bed3cd365653e
I20260927 15:12:28.363827 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:28.375003 23007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:28.389829 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.365450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.97.60:38595)
0927 15:12:28.366312 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.366328 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.366381 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:12:28.369115 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.369149 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.369759 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.370287 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.371749 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.374003 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.375439 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.376673 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.376882 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.381527 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.381539 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.381554 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.381601 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.385316 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.386076 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.386086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.386095 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.386315 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.386703 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.386708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.386712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.387445 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.387680 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.388254 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.388745 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":362,"threads_started":1}
I20260927 15:12:28.395167 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.378473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.97.62:33761)
0927 15:12:28.378699 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.378703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.379588 (+ 885us) server_negotiation.cc:407] Connection header received
0927 15:12:28.379824 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.379830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.379919 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.380032 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.381276 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.382400 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.384222 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.384887 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.385029 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.388803 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.388808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.388811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.388847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.391942 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.392772 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.392777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.392779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.392911 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.393346 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.393351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.393354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.393818 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.393979 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.394541 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.394753 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:12:28.395217 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.378503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33761 (local address 127.0.0.1:48296)
0927 15:12:28.379430 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.379478 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:28.379684 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:28.380094 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:28.380102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:28.380824 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:28.381104 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.381113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.382603 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.382607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.384064 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.384077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.385072 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.385076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.385206 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.385895 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:28.385911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:28.388601 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:28.392079 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.392087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.392093 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.392638 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.393025 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.393031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.393035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.393207 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.393959 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:28.393965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:28.394135 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 15:12:28.394417 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 15:12:28.394745 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":464,"threads_started":1}
I20260927 15:12:28.396687 23007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:28.410475 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.393303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.0.97.62:33761)
0927 15:12:28.393947 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.393952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.393973 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:12:28.394022 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.394027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.394093 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.394209 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.395717 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.396818 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.398567 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.399054 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.399210 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.403788 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.403794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.403797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.403830 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.407322 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.408033 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.408040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.408044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.408198 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.408691 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.408694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.408696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.409020 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.409355 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.409777 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.410019 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":278,"threads_started":1}
I20260927 15:12:28.411279 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.397518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37019 (local address 127.0.0.1:39334)
0927 15:12:28.397791 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.397806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:28.397890 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:28.398276 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:28.398280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:28.398705 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:28.398863 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.398871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.399950 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.399956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.400845 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.400855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.401845 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.401849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.401992 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.402570 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:28.402586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:28.404775 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:28.408380 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.408386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.408390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.408897 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.409337 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.409348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.409352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.409532 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.410236 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:28.410241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:28.410409 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 15:12:28.410573 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 15:12:28.410880 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260927 15:12:28.411548 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.397515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.0.97.61:37019)
0927 15:12:28.397679 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.397683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.397881 (+ 198us) server_negotiation.cc:407] Connection header received
0927 15:12:28.397977 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.397983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.398056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.398156 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.399013 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.399823 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.400995 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.401695 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.401839 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.404975 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.404981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.404986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.405028 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.408217 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.409020 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.409026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.409029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.409199 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.409674 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.409679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.409681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.410033 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.410174 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.410781 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.411127 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:12:28.412680 23007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:28.426640 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.413467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57636)
0927 15:12:28.413672 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.413685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:28.413768 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:28.414085 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:28.414088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:28.414438 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:28.414568 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.414575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.415746 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.415752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.416662 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.416682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.417573 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.417581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.417804 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.418405 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:28.418421 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:28.420393 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:28.424074 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.424079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.424084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.424628 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.425110 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.425116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.425120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.425271 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.425810 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:28.425815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:28.425978 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 15:12:28.426114 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 15:12:28.426368 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 15:12:28.427042 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.413449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.60:38595)
0927 15:12:28.413648 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.413656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.413733 (+ 77us) server_negotiation.cc:407] Connection header received
0927 15:12:28.413848 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.413855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.413938 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.414054 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.414713 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.415615 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.416834 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.417396 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.417499 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.420591 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.420599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.420607 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.420654 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.423944 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.424846 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.424851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.424854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.424990 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.425393 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.425396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.425399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.425690 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.425834 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.426250 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.426571 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":67}
I20260927 15:12:28.429124 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.412611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.0.97.61:37019)
0927 15:12:28.412770 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.412777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.412886 (+ 109us) server_negotiation.cc:407] Connection header received
0927 15:12:28.413070 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.413076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.413148 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.413253 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.414889 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.415746 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.417072 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.417538 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.417668 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.422720 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.422726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.422729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.422763 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.425871 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.426829 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.426835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.426840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.427010 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.427528 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.427534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.427538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.427911 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.428124 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.428501 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.428743 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:12:28.433027 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: Bootstrap starting.
I20260927 15:12:28.434051 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:28.436724 23007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:28.437834 23007 log.cc:824] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:28.439194 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 4
I20260927 15:12:28.436769 22833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48298
I20260927 15:12:28.441282 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: No bootstrap required, opened a new log
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:12:28.452211 23007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.452730 23007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80408fd1e20c4cfdbe52c637cfd2e014, State: Initialized, Role: FOLLOWER
I20260927 15:12:28.453243 23007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.454337 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.454777 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.455758 23007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:28.456158 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.440026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.0.97.60:38595)
0927 15:12:28.440243 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.440251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.440275 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:12:28.440354 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.440359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.440433 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.440563 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.441589 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.442721 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.444573 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.445346 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.445489 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.449011 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.449018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.449023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.449064 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.452807 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.453694 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.453702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.453709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.453934 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.454396 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.454401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.454404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.454895 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.455110 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.455549 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.455787 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:12:28.456187 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.439964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57644)
0927 15:12:28.440127 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.440146 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:28.440256 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:28.440609 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:28.440616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:28.441219 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:28.441418 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.441432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.442885 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.442891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.444404 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.444423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.445505 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.445511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.445719 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.446418 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:28.446436 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:28.448824 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:28.452968 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.452973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.452978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.453567 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.454061 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.454068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.454073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.454268 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.455045 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:28.455052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:28.455268 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 15:12:28.455425 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:12:28.455779 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:12:28.462945 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d: Bootstrap starting.
W20260927 15:12:28.466295 23025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:12:28.466530 23025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:12:28.467267 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:28.468664 22858 log.cc:824] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:28.472492 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d: No bootstrap required, opened a new log
I20260927 15:12:28.463833 22940 master_runner.cc:186] Error getting master registration for 127.0.97.62:33761: OK, instance_id { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" instance_seqno: 1790521947286633 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:12:28.475401 22940 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:12:28.478749 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.479156 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fba176a53c46a0bcb0ce66b8a6f11d, State: Initialized, Role: FOLLOWER
I20260927 15:12:28.479600 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.480535 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.480914 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.481789 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:28.481895 22833 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:12:28.482110 22833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:28.485955 22833 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.489882 22982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "80408fd1e20c4cfdbe52c637cfd2e014"
I20260927 15:12:28.490140 22833 leader_election.cc:288] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [CANDIDATE]: Term 1 election: Requested vote from peers 6ea3ee92abe5412ebc891c11347c5955 (127.0.97.61:37019), 80408fd1e20c4cfdbe52c637cfd2e014 (127.0.97.60:38595)
I20260927 15:12:28.490463 22982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:28.495026 22982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1fba176a53c46a0bcb0ce66b8a6f11d in term 1.
I20260927 15:12:28.496179 22809 leader_election.cc:302] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80408fd1e20c4cfdbe52c637cfd2e014, d1fba176a53c46a0bcb0ce66b8a6f11d; no voters:
I20260927 15:12:28.496869 23028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:28.502059 23028 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [term 1 LEADER]: Becoming Leader. State: Replica: d1fba176a53c46a0bcb0ce66b8a6f11d, State: Running, Role: LEADER
I20260927 15:12:28.502988 23028 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
W20260927 15:12:28.525329 22812 tablet.cc:2369] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:12:28.545317 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.516634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.97.60:38595)
0927 15:12:28.516850 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.516858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.520850 (+ 3992us) server_negotiation.cc:407] Connection header received
0927 15:12:28.521935 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.521943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.522055 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.522147 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.526574 (+ 4427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.527984 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.530438 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.531437 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.531571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.536006 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.536015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.536024 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.536092 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.540622 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.542160 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.542169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.542174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.542383 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.542939 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.542954 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.542961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.543445 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.543737 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.544468 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.544794 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:12:28.563696 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1fba176a53c46a0bcb0ce66b8a6f11d. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.564268 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:28.565353 23048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:28.569631 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } attempt: 5
Sep 27 15:12:28 dist-test-slave-g0w1 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:12:28.579484 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.518486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.97.62:33761)
0927 15:12:28.518686 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.518695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.548782 (+ 30087us) server_negotiation.cc:407] Connection header received
0927 15:12:28.549084 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.549090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.549224 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.549311 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.551722 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.553097 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.555426 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.556384 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.556512 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.562819 (+ 6307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.562826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.562830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.562882 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.568149 (+ 5267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.572700 (+ 4551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.572706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.572710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.572912 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.575142 (+ 2230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.575148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.575151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.575545 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.575812 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.578803 (+ 2991us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.579121 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:12:28.582733 23048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:28.591053 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.570245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38595 (local address 127.0.0.1:57664)
0927 15:12:28.570633 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:28.570655 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:28.570780 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:28.571147 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:28.571152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:28.571653 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:28.571871 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.571885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.573252 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.573259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.574735 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:28.574752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.575912 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.575918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:28.576149 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.576851 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:28.576874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:28.580192 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:28.585204 (+ 5012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.585210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.585216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.585827 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.586670 (+ 843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:28.586676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:28.586680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:28.586983 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:28.588278 (+ 1295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:28.588285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:28.588633 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 15:12:28.588938 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 15:12:28.589743 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
W20260927 15:12:28.591204 22961 tablet.cc:2369] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:12:28.592872 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.570611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.60:38595)
0927 15:12:28.570783 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.570789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.570808 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:12:28.570891 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.570896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.570975 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.571147 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.572017 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.573115 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.574913 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.575642 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.575753 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.580501 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.580510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.580515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.580565 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.584944 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.586050 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.586056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.586059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.586459 (+ 400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.587296 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.587332 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.587340 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.588065 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.589816 (+ 1751us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.590359 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.592411 (+ 2052us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:12:28.594143 23050 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:12:28.599303 22982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [term 1 FOLLOWER]: Refusing update from remote peer d1fba176a53c46a0bcb0ce66b8a6f11d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:12:28.600474 23028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:12:28.608472 22932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: Bootstrap starting.
I20260927 15:12:28.617636 22932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:28.604480 22791 master_runner.cc:186] Error getting master registration for 127.0.97.61:37019: OK, instance_id { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" instance_seqno: 1790521947830683 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:12:28.618724 22791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:12:28.620595 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1fba176a53c46a0bcb0ce66b8a6f11d. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.620944 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.621459 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.519945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.0.97.61:37019)
0927 15:12:28.520194 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.520204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.560026 (+ 39822us) server_negotiation.cc:407] Connection header received
0927 15:12:28.560342 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.560349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.560451 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.560537 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.566433 (+ 5896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.567643 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.581887 (+ 14244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.582641 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.582780 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:28.586416 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:28.586424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:28.586429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:28.586474 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:28.590258 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.591394 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.591399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.591402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.591602 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:28.592267 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:28.592272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:28.592274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:28.592657 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:28.592915 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:28.620764 (+ 27849us) server_negotiation.cc:299] Negotiation successful
0927 15:12:28.621076 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:12:28.622715 22932 log.cc:824] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:28.641281 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.641587 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.644639 22932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: No bootstrap required, opened a new log
I20260927 15:12:28.654886 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.655243 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:28.662146 23048 catalog_manager.cc:1347] Generated new cluster ID: c61dfe8f5ce5431c8e077eff5be93c7b
I20260927 15:12:28.662292 23048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:28.669660 22932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.670406 22932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea3ee92abe5412ebc891c11347c5955, State: Initialized, Role: FOLLOWER
I20260927 15:12:28.671340 22932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } }
I20260927 15:12:28.691757 23057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.695281 22906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:28.692713 23057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.698925 22906 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [term 1 FOLLOWER]: Refusing update from remote peer d1fba176a53c46a0bcb0ce66b8a6f11d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:12:28.699642 22907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ea3ee92abe5412ebc891c11347c5955"
I20260927 15:12:28.702937 22907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1fba176a53c46a0bcb0ce66b8a6f11d in term 1.
I20260927 15:12:28.703790 23028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:12:28.704197 22932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:28.732528 23048 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:28.734941 23048 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:28.748847 23057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1fba176a53c46a0bcb0ce66b8a6f11d. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.749261 23057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.748983 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.749457 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:28.751046 23048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d: Generated new TSK 0
I20260927 15:12:28.752614 23048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:28.754590 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fba176a53c46a0bcb0ce66b8a6f11d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33761 } } peers { permanent_uuid: "6ea3ee92abe5412ebc891c11347c5955" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37019 } } peers { permanent_uuid: "80408fd1e20c4cfdbe52c637cfd2e014" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38595 } } }
I20260927 15:12:28.755093 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1fba176a53c46a0bcb0ce66b8a6f11d [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:28.808089 23073 catalog_manager.cc:1259] Loaded cluster ID: c61dfe8f5ce5431c8e077eff5be93c7b
I20260927 15:12:28.808236 23073 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: loading cluster ID for follower catalog manager: success
I20260927 15:12:28.812835 23073 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: acquiring CA information for follower catalog manager: success
I20260927 15:12:28.816201 23073 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ea3ee92abe5412ebc891c11347c5955: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:12:28.885411 388 mini_kdc.cc:348] Time spent kdestroy: real 0.112s user 0.000s sys 0.022s
W20260927 15:12:28.945361 23012 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:28.930449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.97.62:33761)
0927 15:12:28.930611 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.930618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.935720 (+ 5102us) server_negotiation.cc:407] Connection header received
0927 15:12:28.943533 (+ 7813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.943540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.943665 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.943767 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.945038 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48312: BlockingRecv error: recv got EOF from 127.0.0.1:48312 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 15:12:28.948233 22938 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:28.930051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.0.97.61:37019)
0927 15:12:28.930432 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.930440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.946398 (+ 15958us) server_negotiation.cc:407] Connection header received
0927 15:12:28.946684 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.946690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.946788 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.946862 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.947966 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39344: BlockingRecv error: recv got EOF from 127.0.0.1:39344 (error 108)
Metrics: {"server-negotiator.queue_time_us":218}
W20260927 15:12:28.950758 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:28.936507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.97.60:38595)
0927 15:12:28.936764 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.936771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.949075 (+ 12304us) server_negotiation.cc:407] Connection header received
0927 15:12:28.949323 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.949330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.949413 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.949481 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:28.950485 (+ 1004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57678: BlockingRecv error: recv got EOF from 127.0.0.1:57678 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:12:29.001881 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:28.982291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.97.61:37019)
0927 15:12:28.982489 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:28.982496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:28.993219 (+ 10723us) server_negotiation.cc:407] Connection header received
0927 15:12:28.993852 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:28.993858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:28.994084 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:28.994245 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:28.995026 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.996188 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.998393 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:28.999115 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:28.999237 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.000132 (+ 895us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.000945 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.001086 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.001368 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.001484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 15:12:29.047694 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.037487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.0.97.60:38595)
0927 15:12:29.037723 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.037730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.045258 (+ 7528us) server_negotiation.cc:407] Connection header received
0927 15:12:29.045930 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.045937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.046122 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.046211 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.047403 (+ 1192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57686: BlockingRecv error: recv got EOF from 127.0.0.1:57686 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:12:29.099246 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.079867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.0.97.61:37019)
0927 15:12:29.080106 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.080113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.090887 (+ 10774us) server_negotiation.cc:407] Connection header received
0927 15:12:29.091543 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.091550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.091659 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.091744 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.092531 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.093657 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.096456 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.097152 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.097282 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.098122 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.098310 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.098388 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.098736 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.098837 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:12:29.144747 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.134987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.0.97.60:38595)
0927 15:12:29.135199 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.135206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.142681 (+ 7475us) server_negotiation.cc:407] Connection header received
0927 15:12:29.143207 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.143214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.143313 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.143385 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.144479 (+ 1094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57698: BlockingRecv error: recv got EOF from 127.0.0.1:57698 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:12:29.200462 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.182568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.0.97.61:37019)
0927 15:12:29.182793 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.182801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.193599 (+ 10798us) server_negotiation.cc:407] Connection header received
0927 15:12:29.194125 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.194132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.194226 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.194296 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.195086 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.196128 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.198036 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.198695 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.198834 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.199564 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.199710 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.199788 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.200006 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.200094 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 15:12:29.210376 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.208050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.0.97.60:38595)
0927 15:12:29.208288 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.208298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.208855 (+ 557us) server_negotiation.cc:407] Connection header received
0927 15:12:29.209096 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.209107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.209184 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.209309 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.210107 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57710: BlockingRecv error: recv got EOF from 127.0.0.1:57710 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:12:29.224120 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.217982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.97.61:37019)
0927 15:12:29.218305 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.218316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.218859 (+ 543us) server_negotiation.cc:407] Connection header received
0927 15:12:29.219108 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.219114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.219197 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.219310 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.219743 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.220531 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.222002 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.222609 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.222772 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.223224 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.223386 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.223518 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.223724 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.223811 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
W20260927 15:12:29.232669 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.230618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.0.97.60:38595)
0927 15:12:29.230915 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.230921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.231266 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:12:29.231443 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.231449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.231555 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.231681 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.232436 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57712: BlockingRecv error: recv got EOF from 127.0.0.1:57712 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
I20260927 15:12:29.250773 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.243930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.0.97.61:37019)
0927 15:12:29.244164 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.244172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.244595 (+ 423us) server_negotiation.cc:407] Connection header received
0927 15:12:29.244813 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.244817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.244887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.244968 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.245492 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.246432 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.248492 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.249165 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.249333 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.249940 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.250055 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.250198 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.250324 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.250430 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:12:29.270006 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.267950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.0.97.60:38595)
0927 15:12:29.268160 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.268167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.268473 (+ 306us) server_negotiation.cc:407] Connection header received
0927 15:12:29.268650 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.268655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.268773 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.268908 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.269739 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57716: BlockingRecv error: recv got EOF from 127.0.0.1:57716 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:12:29.297266 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.289877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.97.61:37019)
0927 15:12:29.290091 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.290098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.290454 (+ 356us) server_negotiation.cc:407] Connection header received
0927 15:12:29.290624 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.290631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.290736 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.290881 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.291496 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.292762 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.294851 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.295604 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.295802 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.296342 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.296524 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.296626 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.296855 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.296951 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:12:29.308095 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.305712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.97.60:38595)
0927 15:12:29.305967 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.305975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.306486 (+ 511us) server_negotiation.cc:407] Connection header received
0927 15:12:29.306719 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.306726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.306830 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.306937 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.307864 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57724: BlockingRecv error: recv got EOF from 127.0.0.1:57724 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 15:12:29.350404 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.343798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.97.61:37019)
0927 15:12:29.344059 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.344067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.344614 (+ 547us) server_negotiation.cc:407] Connection header received
0927 15:12:29.344796 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.344800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.344898 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.344985 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.345549 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.346342 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.348440 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.348978 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.349095 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.349617 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.349729 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.349832 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.349979 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.350054 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 15:12:29.359949 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.357545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.0.97.60:38595)
0927 15:12:29.357775 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.357800 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.358370 (+ 570us) server_negotiation.cc:407] Connection header received
0927 15:12:29.358576 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.358582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.358700 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.358807 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.359722 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57736: BlockingRecv error: recv got EOF from 127.0.0.1:57736 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:12:29.433298 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.427410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.97.61:37019)
0927 15:12:29.427717 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.427725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.428054 (+ 329us) server_negotiation.cc:407] Connection header received
0927 15:12:29.428311 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.428320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.428427 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.428534 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.428986 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.429852 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.431483 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.431956 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.432111 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.432522 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.432646 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.432758 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.432911 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.432998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260927 15:12:29.441556 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:29.439504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.0.97.60:38595)
0927 15:12:29.439733 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.439741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.440124 (+ 383us) server_negotiation.cc:407] Connection header received
0927 15:12:29.440365 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.440369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.440444 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.440552 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:29.441377 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51658: BlockingRecv error: recv got EOF from 127.0.0.1:51658 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:12:29.468604 23025 catalog_manager.cc:1259] Loaded cluster ID: c61dfe8f5ce5431c8e077eff5be93c7b
I20260927 15:12:29.468767 23025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: loading cluster ID for follower catalog manager: success
I20260927 15:12:29.471402 23025 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: acquiring CA information for follower catalog manager: success
I20260927 15:12:29.473361 23025 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80408fd1e20c4cfdbe52c637cfd2e014: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:12:29.578980 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.573657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.0.97.61:37019)
0927 15:12:29.573963 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.573979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.574204 (+ 225us) server_negotiation.cc:407] Connection header received
0927 15:12:29.574458 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.574462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.574533 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.574643 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.575125 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.575825 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.577179 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.577632 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.577789 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.578224 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.578338 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.578440 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.578593 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.578683 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 15:12:29.591367 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.585339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.0.97.60:38595)
0927 15:12:29.585576 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.585586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.585970 (+ 384us) server_negotiation.cc:407] Connection header received
0927 15:12:29.586122 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.586126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.586303 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.586410 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.586881 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.587664 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.589038 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.589471 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.589570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.590096 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.590657 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.590802 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.590895 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.590965 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:12:29.610267 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.598473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.97.62:33761)
0927 15:12:29.603678 (+ 5205us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.603685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.603703 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:12:29.603832 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.603840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.604091 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.604301 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.605192 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.606353 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.608056 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.608472 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.608612 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.609131 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.609499 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.609626 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.609812 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.609903 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5058,"thread_start_us":4959,"threads_started":1}
I20260927 15:12:29.611205 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.598447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.0.97.60:38595)
0927 15:12:29.598713 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.598721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.603426 (+ 4705us) server_negotiation.cc:407] Connection header received
0927 15:12:29.603724 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.603729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.603815 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.603928 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.604779 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.605901 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.608633 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.609354 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.609455 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.610360 (+ 905us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.610493 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.610610 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.610750 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.610861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 15:12:29.611541 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:29.598366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.0.97.61:37019)
0927 15:12:29.598604 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:29.598613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:29.603197 (+ 4584us) server_negotiation.cc:407] Connection header received
0927 15:12:29.603688 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:29.603696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:29.603800 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:29.603906 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:29.604943 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.606099 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.608634 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:29.609354 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:29.609455 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:29.610740 (+ 1285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:29.610868 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:29.610959 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:29.611194 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 15:12:29.611290 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:12:29.616137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22791
I20260927 15:12:29.617379 22853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:29.730707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22791
W20260927 15:12:29.751499 22882 connection.cc:535] server connection from 127.0.0.1:39320 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:12:29.751852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22865
I20260927 15:12:29.753142 22927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:29.865979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22865
I20260927 15:12:29.887022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 22940
I20260927 15:12:29.888264 23002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:29.999804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 22940
2026-09-27T15:12:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4690 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:30 dist-test-slave-g0w1 krb5kdc[23187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:30 dist-test-slave-g0w1 krb5kdc[23187](info): set up 2 sockets
Sep 27 15:12:30 dist-test-slave-g0w1 krb5kdc[23187](info): commencing operation
krb5kdc: starting...
W20260927 15:12:31.131770 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/test-user.keytab.
Sep 27 15:12:31 dist-test-slave-g0w1 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 15:12:31.415218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45889
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45889,127.0.97.61:37319,127.0.97.60:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:31.772564 23200 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:12:31.772971 23200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:31.773072 23200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:31.773232 23200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:31.780841 23200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:31.780953 23200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:31.781013 23200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:31.781104 23200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:31.781200 23200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:31.792246 23200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45889,127.0.97.61:37319,127.0.97.60:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:31.793690 23200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:31.795142 23200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:31.808212 23208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:31.808212 23206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:31.808365 23205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:31.809128 23200 server_base.cc:1034] running on GCE node
I20260927 15:12:31.810030 23200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:31.811832 23200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:31.813102 23200 hybrid_clock.cc:648] HybridClock initialized: now 1790521951813028 us; error 41 us; skew 500 ppm
I20260927 15:12:31.813529 23200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:31 dist-test-slave-g0w1 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:31.827337 23200 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 15:12:31.829228 23200 webserver.cc:460] Webserver started at http://127.0.97.62:36299/ using document root <none> and password file <none>
I20260927 15:12:31.830075 23200 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:31.830235 23200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:31.830582 23200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:31.834046 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data/instance:
uuid: "5372ce963d494f678e6d1fa1804e4f3c"
format_stamp: "Formatted at 2026-09-27 15:12:31 on dist-test-slave-g0w1"
server_key: "a5d448af44da95a3bdc11c3340512347"
server_key_iv: "97aa30836f4bc7d196922cdcc4cf43ec"
server_key_version: "encryptionkey@0"
I20260927 15:12:31.835033 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal/instance:
uuid: "5372ce963d494f678e6d1fa1804e4f3c"
format_stamp: "Formatted at 2026-09-27 15:12:31 on dist-test-slave-g0w1"
server_key: "a5d448af44da95a3bdc11c3340512347"
server_key_iv: "97aa30836f4bc7d196922cdcc4cf43ec"
server_key_version: "encryptionkey@0"
I20260927 15:12:31.841325 23200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:12:31.845619 23215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:31.847476 23200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:12:31.847699 23200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal
uuid: "5372ce963d494f678e6d1fa1804e4f3c"
format_stamp: "Formatted at 2026-09-27 15:12:31 on dist-test-slave-g0w1"
server_key: "a5d448af44da95a3bdc11c3340512347"
server_key_iv: "97aa30836f4bc7d196922cdcc4cf43ec"
server_key_version: "encryptionkey@0"
I20260927 15:12:31.847955 23200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:31.872136 23200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:31.877620 23200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:31.877954 23200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:31.898491 23200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45889
I20260927 15:12:31.898514 23266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45889 every 8 connection(s)
I20260927 15:12:31.900054 23200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:31.904063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 23200
I20260927 15:12:31.904258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-0/wal/instance
I20260927 15:12:31.904359 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:31.904836 388 external_mini_cluster.cc:1428] Setting key 8ffe62856ef0bf8997eb36196a7b096d
I20260927 15:12:31.915371 23267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 15:12:31 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:31.948925 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:31.919826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.97.62:45889)
0927 15:12:31.920750 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:31.920762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:31.923921 (+ 3159us) server_negotiation.cc:407] Connection header received
0927 15:12:31.926218 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:31.926225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:31.926524 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:31.926770 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:31.928265 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:31.929543 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:31.931163 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:31.931773 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:31.932019 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:31.938499 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:31.938514 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:31.938538 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:31.938594 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:31.943023 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:31.944151 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:31.944164 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:31.944171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:31.944413 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:31.944960 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:31.944969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:31.944975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:31.945822 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:31.946049 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:31.946792 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 15:12:31.947396 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":271,"server-negotiator.queue_time_us":700,"thread_start_us":465,"threads_started":1}
I20260927 15:12:31.949071 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:31.919826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45889 (local address 127.0.0.1:34194)
0927 15:12:31.923706 (+ 3880us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:31.923759 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:31.925748 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:31.926777 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:31.926785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:31.927647 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:31.928024 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:31.928044 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:31.929712 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:31.929719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:31.930969 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:31.930982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:31.932040 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:31.932047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:31.932208 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:31.932938 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:31.932962 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:31.938090 (+ 5128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:31.943312 (+ 5222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:31.943327 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:31.943339 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:31.943936 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:31.944543 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:31.944550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:31.944555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:31.944795 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:31.945978 (+ 1183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:31.945989 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:31.946298 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 15:12:31.946575 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 15:12:31.947262 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3611,"thread_start_us":3375,"threads_started":1}
I20260927 15:12:31.952673 23267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:31.954221 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:31.953594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52436)
0927 15:12:31.953894 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:31.954018 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 15:12:31.954571 23218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
I20260927 15:12:31.956609 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:31.956175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52438)
0927 15:12:31.956359 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:31.956451 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:12:31.956970 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 15:12:31.979816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37319
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45889,127.0.97.61:37319,127.0.97.60:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:12:32.006405 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } attempt: 1
I20260927 15:12:32.007669 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.007221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52444)
0927 15:12:32.007419 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.007521 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 15:12:32.008884 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.008496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52450)
0927 15:12:32.008654 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.008739 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:12:32.009125 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
I20260927 15:12:32.077448 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } attempt: 2
I20260927 15:12:32.078727 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.078194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52454)
0927 15:12:32.078490 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.078579 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 15:12:32.080135 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.079655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52460)
0927 15:12:32.079807 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.079887 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:12:32.080338 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
I20260927 15:12:32.171685 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } attempt: 3
I20260927 15:12:32.172899 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.172421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52468)
0927 15:12:32.172647 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.172751 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 15:12:32.174288 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.173798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52482)
0927 15:12:32.173987 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.174079 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 15:12:32.174589 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
I20260927 15:12:32.317896 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } attempt: 4
I20260927 15:12:32.319312 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.318603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52484)
0927 15:12:32.318952 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.319126 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260927 15:12:32.320559 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.320067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52490)
0927 15:12:32.320283 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.320395 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 15:12:32.320832 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37319: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37319: connect: Connection refused (error 111)
W20260927 15:12:32.336498 23274 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:12:32.337181 23274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:32.337306 23274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:32.337458 23274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:32.345504 23274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:32.345618 23274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:32.345709 23274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:32.345788 23274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:32.345884 23274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:32.356702 23274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45889,127.0.97.61:37319,127.0.97.60:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:32.358124 23274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:32.359609 23274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:32.372467 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:32.372541 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:32.372486 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:32.373585 23274 server_base.cc:1034] running on GCE node
I20260927 15:12:32.374462 23274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:32.376096 23274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:32.377378 23274 hybrid_clock.cc:648] HybridClock initialized: now 1790521952377321 us; error 61 us; skew 500 ppm
I20260927 15:12:32.377791 23274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:32 dist-test-slave-g0w1 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:32.381361 23274 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 15:12:32.382995 23274 webserver.cc:460] Webserver started at http://127.0.97.61:43611/ using document root <none> and password file <none>
I20260927 15:12:32.383796 23274 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:32.383893 23274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:32.384166 23274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:32.387499 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data/instance:
uuid: "515c6c7460fc4db7ae408f7c64d0bcde"
format_stamp: "Formatted at 2026-09-27 15:12:32 on dist-test-slave-g0w1"
server_key: "3d83360b59608d7d6f88dc7902f44d95"
server_key_iv: "7cf272eb8763ac13ece536815f946914"
server_key_version: "encryptionkey@0"
I20260927 15:12:32.388386 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal/instance:
uuid: "515c6c7460fc4db7ae408f7c64d0bcde"
format_stamp: "Formatted at 2026-09-27 15:12:32 on dist-test-slave-g0w1"
server_key: "3d83360b59608d7d6f88dc7902f44d95"
server_key_iv: "7cf272eb8763ac13ece536815f946914"
server_key_version: "encryptionkey@0"
I20260927 15:12:32.394253 23274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:12:32.398574 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:32.400424 23274 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:12:32.400614 23274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal
uuid: "515c6c7460fc4db7ae408f7c64d0bcde"
format_stamp: "Formatted at 2026-09-27 15:12:32 on dist-test-slave-g0w1"
server_key: "3d83360b59608d7d6f88dc7902f44d95"
server_key_iv: "7cf272eb8763ac13ece536815f946914"
server_key_version: "encryptionkey@0"
I20260927 15:12:32.400831 23274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:32.415977 23274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:32.420558 23274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:32.420807 23274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:32.440589 23274 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37319
I20260927 15:12:32.440596 23340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37319 every 8 connection(s)
I20260927 15:12:32.442021 23274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/data/info.pb
I20260927 15:12:32.446030 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:32.447464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 23274
I20260927 15:12:32.447657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-1/wal/instance
I20260927 15:12:32.448225 388 external_mini_cluster.cc:1428] Setting key 17a91c21734aa75745a2f65328de67bf
I20260927 15:12:32.456604 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:32 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 15:12:32.484782 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.460421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.0.97.62:45889)
0927 15:12:32.461001 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:32.461008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:32.462126 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 15:12:32.465313 (+ 3187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:32.465318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:32.465399 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:32.465507 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:32.468019 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.468839 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.471581 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.472039 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.472166 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:32.477326 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:32.477331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:32.477335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:32.477378 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:32.480548 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:32.481627 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:32.481643 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:32.481646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:32.481794 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:32.482456 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:32.482465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:32.482471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:32.482912 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:32.483181 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:32.483940 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 15:12:32.484252 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":296,"threads_started":1}
I20260927 15:12:32.485522 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.460986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45889 (local address 127.0.0.1:34200)
0927 15:12:32.461909 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.461968 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:32.464916 (+ 2948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:32.465852 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:32.465869 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:32.466766 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:32.467727 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:32.467757 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.469040 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.469049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:32.471379 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:32.471402 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.472322 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.472338 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:32.472721 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:32.473590 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:32.473633 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:32.476979 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:32.480756 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:32.480765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:32.480782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:32.481447 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:32.481958 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:32.481963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:32.481966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:32.482256 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:32.483063 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:32.483116 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:32.483512 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 15:12:32.483767 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 15:12:32.484230 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":340,"threads_started":1}
I20260927 15:12:32.487591 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:32 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:32.505944 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.488511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52494)
0927 15:12:32.488858 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.488879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:32.489021 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:32.490083 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:32.490086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:32.490524 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:32.490710 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:32.490719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.492572 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.492588 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:32.494398 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:32.494422 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.495481 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.495496 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:32.495751 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:32.496547 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:32.496580 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:32.499484 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:32.503023 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:32.503030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:32.503034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:32.503484 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:32.503906 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:32.503912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:32.503916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:32.504062 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:32.504754 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:32.504764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:32.504935 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 15:12:32.505077 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:12:32.505414 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 15:12:32.505983 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.488741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.97.61:37319)
0927 15:12:32.489509 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:32.489521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:32.489559 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:12:32.489624 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:32.489635 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:32.489907 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:32.490092 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:32.490894 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.492367 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.494604 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.495311 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.495420 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:32.499739 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:32.499757 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:32.499794 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:32.499827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:32.502844 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:32.503650 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:32.503657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:32.503660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:32.503787 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:32.504172 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:32.504179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:32.504182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:32.504622 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:32.504856 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:32.505248 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 15:12:32.505541 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":568,"thread_start_us":410,"threads_started":1}
I20260927 15:12:32.509104 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:32.510423 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.509903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52828)
0927 15:12:32.510102 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.510219 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 15:12:32.510707 23292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.512518 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.512133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52830)
0927 15:12:32.512310 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.512397 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:12:32.512789 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 15:12:32.518556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0r1zNn/build/asan/bin/kudu
/tmp/dist-test-task0r1zNn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34501
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45889,127.0.97.61:37319,127.0.97.60:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:12:32.562176 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 1
I20260927 15:12:32.563417 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.562930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52840)
0927 15:12:32.563145 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.563248 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 15:12:32.564513 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.564154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52844)
0927 15:12:32.564324 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.564406 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:12:32.564786 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.620101 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } attempt: 5
Sep 27 15:12:32 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:32.633054 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 2
I20260927 15:12:32.634342 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.633760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52856)
0927 15:12:32.633976 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.634080 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 15:12:32.635473 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.635152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52866)
0927 15:12:32.635291 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.635354 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:12:32.635941 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.636193 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.620920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52496)
0927 15:12:32.621132 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.621155 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:32.621286 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:32.621750 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:32.621754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:32.622180 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:32.622399 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:32.622410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.623930 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.623934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:32.624845 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:32.624858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.625913 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.625918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:32.626063 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:32.626775 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:32.626795 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:32.629194 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:32.633198 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:32.633204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:32.633208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:32.633749 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:32.634262 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:32.634268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:32.634271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:32.634454 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:32.635168 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:32.635173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:32.635363 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 15:12:32.635526 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:12:32.635871 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 15:12:32.636231 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.621029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.0.97.61:37319)
0927 15:12:32.621241 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:32.621247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:32.621272 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:12:32.621451 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:32.621457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:32.621544 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:32.621709 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:32.622599 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.623734 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.625040 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:32.625728 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:32.625894 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:32.629449 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:32.629456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:32.629460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:32.629532 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:32.632987 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:32.633945 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:32.633950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:32.633954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:32.634104 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:32.634580 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:32.634585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:32.634588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:32.635015 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:32.635186 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:32.635696 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 15:12:32.635948 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 15:12:32.637640 23267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:32.638798 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.638333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52876)
0927 15:12:32.638515 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.638611 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:12:32.639983 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.639495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52888)
0927 15:12:32.639694 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.639801 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:12:32.640302 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.691565 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 1
I20260927 15:12:32.693140 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.692463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52896)
0927 15:12:32.692755 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.692891 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 15:12:32.694525 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.694035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52904)
0927 15:12:32.694236 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.694351 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:12:32.694864 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.727185 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 3
I20260927 15:12:32.728458 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.727906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52920)
0927 15:12:32.728152 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.728290 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 15:12:32.729907 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.729424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52926)
0927 15:12:32.729686 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.729775 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260927 15:12:32.730232 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.763161 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 2
I20260927 15:12:32.764532 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.763875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52928)
0927 15:12:32.764150 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.764312 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 15:12:32.765712 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.765278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52944)
0927 15:12:32.765482 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.765574 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:12:32.765928 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.872218 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 3
W20260927 15:12:32.872603 23349 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:12:32.873018 23349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:32.873142 23349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:32.873291 23349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:12:32.873466 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 4
I20260927 15:12:32.873570 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.872993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52948)
0927 15:12:32.873251 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.873374 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 15:12:32.874600 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.874104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52964)
0927 15:12:32.874323 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.874423 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:12:32.874786 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.874354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52972)
0927 15:12:32.874541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.874630 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:12:32.875138 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
I20260927 15:12:32.875590 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:32.875301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:52988)
0927 15:12:32.875428 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:32.875489 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:12:32.875866 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34501: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34501: connect: Connection refused (error 111)
W20260927 15:12:32.881898 23349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:32.882043 23349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:32.882148 23349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:32.882246 23349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:32.882336 23349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:32.892838 23349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45889,127.0.97.61:37319,127.0.97.60:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:32.894281 23349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:32.895754 23349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:32.910079 23355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:32.910115 23354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:32.910538 23357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:32.911199 23349 server_base.cc:1034] running on GCE node
I20260927 15:12:32.911974 23349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:32.913452 23349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:32.914746 23349 hybrid_clock.cc:648] HybridClock initialized: now 1790521952914614 us; error 82 us; skew 500 ppm
I20260927 15:12:32.915158 23349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:32 dist-test-slave-g0w1 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:32.918593 23349 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 15:12:32.920243 23349 webserver.cc:460] Webserver started at http://127.0.97.60:40749/ using document root <none> and password file <none>
I20260927 15:12:32.921025 23349 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:32.921169 23349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:32.921526 23349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:32.924710 23349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data/instance:
uuid: "f24bc51f6fdb487498d541f4a7bd1f5f"
format_stamp: "Formatted at 2026-09-27 15:12:32 on dist-test-slave-g0w1"
server_key: "1f714a22d18a10359fc32cdf6f41511a"
server_key_iv: "1ebe8d24b374bbfb57df14e2e7acc368"
server_key_version: "encryptionkey@0"
I20260927 15:12:32.925540 23349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal/instance:
uuid: "f24bc51f6fdb487498d541f4a7bd1f5f"
format_stamp: "Formatted at 2026-09-27 15:12:32 on dist-test-slave-g0w1"
server_key: "1f714a22d18a10359fc32cdf6f41511a"
server_key_iv: "1ebe8d24b374bbfb57df14e2e7acc368"
server_key_version: "encryptionkey@0"
I20260927 15:12:32.931028 23349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:12:32.935034 23364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:32.936866 23349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:32.937093 23349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data,/tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal
uuid: "f24bc51f6fdb487498d541f4a7bd1f5f"
format_stamp: "Formatted at 2026-09-27 15:12:32 on dist-test-slave-g0w1"
server_key: "1f714a22d18a10359fc32cdf6f41511a"
server_key_iv: "1ebe8d24b374bbfb57df14e2e7acc368"
server_key_version: "encryptionkey@0"
I20260927 15:12:32.937366 23349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:32.965874 23349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:32.970495 23349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:32.970793 23349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:32.990545 23349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34501
I20260927 15:12:32.990566 23415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34501 every 8 connection(s)
I20260927 15:12:32.992066 23349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/data/info.pb
I20260927 15:12:32.996073 23416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:32.997458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0r1zNn/build/asan/bin/kudu as pid 23349
I20260927 15:12:32.997643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0r1zNn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521577514430-388-0/minicluster-data/master-2/wal/instance
I20260927 15:12:32.998193 388 external_mini_cluster.cc:1428] Setting key 355b6008fba03a1fb5e906f5456b7b30
I20260927 15:12:33.005872 23416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:33.023414 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.000178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.0.97.60:34501)
0927 15:12:33.001043 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.001056 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.001105 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:12:33.002785 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.002811 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.003357 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.003789 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.005270 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.006643 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.008053 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.009261 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.009464 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.014549 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.014558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.014570 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.014609 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.018317 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.019234 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.019244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.019254 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.019450 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.019968 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.019978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.019980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.020575 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.020934 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.021387 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.021875 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":397,"threads_started":1}
I20260927 15:12:33.025275 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.008735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.0.97.62:45889)
0927 15:12:33.009306 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.009314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.009859 (+ 545us) server_negotiation.cc:407] Connection header received
0927 15:12:33.010097 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.010105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.010210 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.010332 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.012052 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.012724 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.014649 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.015142 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.015304 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.018850 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.018855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.018858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.018895 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.022102 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.022980 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.022985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.022988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.023220 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.023724 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.023730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.023733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.024094 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.024267 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.024756 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.024989 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":267,"threads_started":1}
I20260927 15:12:33.025491 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.008773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45889 (local address 127.0.0.1:34210)
0927 15:12:33.009681 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:33.009730 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:33.009928 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:33.010474 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:33.010482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:33.011450 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:33.011863 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.011879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.012914 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.012920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.014481 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.014497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.015331 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.015337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.015524 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.016152 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:33.016177 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:33.018636 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:33.022246 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.022256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.022263 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.022820 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.023401 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.023407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.023412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.023568 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.024258 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:33.024263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:33.024449 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 15:12:33.024606 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:12:33.024932 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":478,"threads_started":1}
I20260927 15:12:33.027346 23416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 15:12:33.043390 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.027992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37319 (local address 127.0.0.1:52500)
0927 15:12:33.028175 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:33.028195 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:33.028302 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:33.028630 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:33.028636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:33.029118 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:33.029333 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.029344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.030863 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.030867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.031854 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.031867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.032661 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.032668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.032907 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.033662 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:33.033683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:33.036208 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:33.040040 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.040049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.040055 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.040662 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.041335 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.041340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.041344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.041563 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.042254 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:33.042259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:33.042463 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 15:12:33.042653 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 15:12:33.042973 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:12:33.043431 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.028099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.0.97.61:37319)
0927 15:12:33.028273 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.028278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.028297 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:12:33.028429 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.028434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.028494 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.028573 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.029544 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.030644 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.032006 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.032492 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.032593 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.036541 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.036549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.036554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.036606 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.039849 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.040877 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.040885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.040889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.041096 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.041732 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.041739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.041744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.042118 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.042292 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.042747 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.042972 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:12:33.044032 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.027070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.97.62:45889)
0927 15:12:33.027242 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.027249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.027331 (+ 82us) server_negotiation.cc:407] Connection header received
0927 15:12:33.027534 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.027539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.027625 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.027727 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.029537 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.030659 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.032122 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.032532 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.032682 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.037586 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.037595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.037602 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.037662 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.040933 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.041803 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.041813 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.041816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.041960 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.042417 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.042421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.042424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.042775 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.043081 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.043467 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.043717 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:12:33.044983 23416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:12:33.052397 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 4
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 15:12:33.064309 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.046253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.97.61:37319)
0927 15:12:33.046418 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.046423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.046440 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:12:33.046494 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.046499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.046571 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.046678 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.048252 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.049315 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.051151 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.051853 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.052000 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.057610 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.057615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.057618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.057656 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.060582 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.061500 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.061512 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.061514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.061646 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.062136 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.062145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.062151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.062698 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.063040 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.063426 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.063767 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:12:33.064513 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.045811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:53010)
0927 15:12:33.045982 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:33.046001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:33.046104 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:33.047226 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:33.047233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:33.047767 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:33.047961 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.047974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.049447 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.049453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.050910 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.050927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.052308 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.052318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.052563 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.053192 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:33.053223 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:33.056621 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:33.061863 (+ 5242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.061869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.061872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.062348 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.062716 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.062723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.062729 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.062908 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.063635 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:33.063640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:33.063786 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 15:12:33.063928 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:12:33.064187 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:12:33.065389 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.046031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.0.97.60:34501)
0927 15:12:33.046451 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.046458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.046473 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:12:33.046514 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.046518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.046576 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.046671 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.048120 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.049255 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.051151 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.051858 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.052002 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.056892 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.056903 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.056911 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.056966 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.061692 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.062473 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.062481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.062486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.062601 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.063032 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.063036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.063038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.063444 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.063583 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.064773 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.065036 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260927 15:12:33.071038 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:12:33.072530 23416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: Bootstrap starting.
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:12:33.073637 23242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5372ce963d494f678e6d1fa1804e4f3c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34214
I20260927 15:12:33.076936 23416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:33.078526 23416 log.cc:824] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:33.082396 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.053249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:53012)
0927 15:12:33.053607 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:33.053626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:33.053848 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:33.066079 (+ 12231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:33.066084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:33.066645 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:33.066917 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.066932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.068582 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.068588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.070076 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.070094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.071163 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.071171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.071382 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.072125 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:33.072153 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:33.075215 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:33.079766 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.079772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.079778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.080190 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.080648 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.080652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.080655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.080813 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.081357 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:33.081362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:33.081516 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 15:12:33.081664 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 15:12:33.082003 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260927 15:12:33.082394 23416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: No bootstrap required, opened a new log
I20260927 15:12:33.082449 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.053360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.0.97.60:34501)
0927 15:12:33.065642 (+ 12282us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.065651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.065690 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:12:33.065751 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.065758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.065850 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.065993 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.067305 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.068422 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.070236 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.070906 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.071063 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.075468 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.075475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.075483 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.075523 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.079570 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.080331 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.080337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.080341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.080502 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.080939 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.080946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.080954 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.081221 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.081375 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.081782 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.081993 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12122,"thread_start_us":12808,"threads_started":1}
I20260927 15:12:33.089210 23267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c: Bootstrap starting.
I20260927 15:12:33.093676 23267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:33.093592 23416 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.094115 23416 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24bc51f6fdb487498d541f4a7bd1f5f, State: Initialized, Role: FOLLOWER
I20260927 15:12:33.094712 23416 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.095239 23267 log.cc:824] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:33.095821 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.096297 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.097232 23416 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:33.099220 23267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c: No bootstrap required, opened a new log
I20260927 15:12:33.108186 23267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.108690 23267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5372ce963d494f678e6d1fa1804e4f3c, State: Initialized, Role: FOLLOWER
I20260927 15:12:33.109285 23267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
W20260927 15:12:33.109505 23436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:12:33.109791 23436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:12:33.111732 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.112282 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.113154 23267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:33.113225 23242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:12:33.113437 23242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:33.106480 23349 master_runner.cc:186] Error getting master registration for 127.0.97.62:45889: OK, instance_id { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" instance_seqno: 1790521951892746 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:12:33.117981 23349 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:12:33.118039 23242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.120268 23242 leader_election.cc:288] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [CANDIDATE]: Term 1 election: Requested vote from peers 515c6c7460fc4db7ae408f7c64d0bcde (127.0.97.61:37319), f24bc51f6fdb487498d541f4a7bd1f5f (127.0.97.60:34501)
I20260927 15:12:33.121380 23391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5372ce963d494f678e6d1fa1804e4f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f"
I20260927 15:12:33.121937 23391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:33.125811 23391 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5372ce963d494f678e6d1fa1804e4f3c in term 1.
I20260927 15:12:33.126590 23218 leader_election.cc:302] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5372ce963d494f678e6d1fa1804e4f3c, f24bc51f6fdb487498d541f4a7bd1f5f; no voters:
I20260927 15:12:33.127198 23438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:33.131363 23438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [term 1 LEADER]: Becoming Leader. State: Replica: 5372ce963d494f678e6d1fa1804e4f3c, State: Running, Role: LEADER
W20260927 15:12:33.131441 23221 tablet.cc:2369] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:12:33.132112 23438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.156641 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.136365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.0.97.62:45889)
0927 15:12:33.136566 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.136573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.137421 (+ 848us) server_negotiation.cc:407] Connection header received
0927 15:12:33.137738 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.137745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.137833 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.137911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.140211 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.141418 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.143547 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.144284 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.144399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.148184 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.148191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.148197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.148251 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.152315 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.153731 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.153738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.153742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.153932 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.154446 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.154453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.154456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.154935 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.155303 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.155927 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.156238 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:12:33.158134 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5372ce963d494f678e6d1fa1804e4f3c. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.158635 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:33.163774 23458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:33.175117 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } attempt: 5
I20260927 15:12:33.175796 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.140361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.0.97.61:37319)
0927 15:12:33.140595 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.140604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.151812 (+ 11208us) server_negotiation.cc:407] Connection header received
0927 15:12:33.152755 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.152765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.152907 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.153010 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.156273 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.157322 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.161325 (+ 4003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.162303 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.162441 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.168350 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.168357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.168362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.168408 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.172337 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.173214 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.173220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.173223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.173426 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.173870 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.173878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.173881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.174276 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.174524 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.175006 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.175410 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:12:33.177796 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.152403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.97.60:34501)
0927 15:12:33.152625 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.152632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.159588 (+ 6956us) server_negotiation.cc:407] Connection header received
0927 15:12:33.159876 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.159883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.159998 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.160078 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.161987 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.163177 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.165137 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.165777 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.165903 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.170319 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.170327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.170332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.170383 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.173901 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.174777 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.174783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.174786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.174949 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.175585 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.175590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.175594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.176008 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.176271 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.176710 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.177159 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:12:33.170590 23200 master_runner.cc:186] Error getting master registration for 127.0.97.61:37319: OK, instance_id { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" instance_seqno: 1790521952435139 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:12:33.182785 23200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 15:12:33 dist-test-slave-g0w1 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 15:12:33.190871 23458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:33.194658 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.175664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34501 (local address 127.0.0.1:53018)
0927 15:12:33.175880 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:33.175900 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:33.176002 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:33.176996 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:33.177002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:33.177523 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:33.177757 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.177772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.179390 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.179396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.180796 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:33.180812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.181565 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.181571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:33.181805 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.182583 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:33.182603 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:33.185700 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:33.191335 (+ 5635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.191341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.191345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.191974 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.192508 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:33.192516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:33.192522 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:33.192720 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:33.193545 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:33.193551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:33.193726 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 15:12:33.193895 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 15:12:33.194241 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 15:12:33.195889 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.176231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.0.97.60:34501)
0927 15:12:33.176663 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.176669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.176686 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:12:33.176751 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.176759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.176849 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.176955 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.177918 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.179017 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.180948 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.181417 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.181515 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.185985 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:33.185997 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:33.186002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:33.186046 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:33.190939 (+ 4893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.192148 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.192154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.192157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.192361 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:33.192883 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:33.192888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:33.192892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:33.193307 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:33.193473 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.195223 (+ 1750us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.195496 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260927 15:12:33.202598 23341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: Bootstrap starting.
I20260927 15:12:33.208382 23341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:33.209978 23341 log.cc:824] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:33.220348 23341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: No bootstrap required, opened a new log
W20260927 15:12:33.223034 23370 tablet.cc:2369] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:12:33.232817 23341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.233403 23341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515c6c7460fc4db7ae408f7c64d0bcde, State: Initialized, Role: FOLLOWER
I20260927 15:12:33.234042 23341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } }
I20260927 15:12:33.238554 23391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [term 1 FOLLOWER]: Refusing update from remote peer 5372ce963d494f678e6d1fa1804e4f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:12:33.240168 23438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:12:33.247581 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.248283 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.249040 23341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:33.249275 23316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5372ce963d494f678e6d1fa1804e4f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "515c6c7460fc4db7ae408f7c64d0bcde"
I20260927 15:12:33.249814 23315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:33.253815 23315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [term 1 FOLLOWER]: Refusing update from remote peer 5372ce963d494f678e6d1fa1804e4f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:12:33.255147 23438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:12:33.257459 23316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5372ce963d494f678e6d1fa1804e4f3c in term 1.
I20260927 15:12:33.264948 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5372ce963d494f678e6d1fa1804e4f3c. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.265327 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.271380 23461 mvcc.cc:205] Tried to move back new op lower bound from 7333977920449634304 to 7333977920043941888. Current Snapshot: MvccSnapshot[applied={T|T < 7333977920449634304}]
I20260927 15:12:33.273765 23464 mvcc.cc:205] Tried to move back new op lower bound from 7333977920449634304 to 7333977920043941888. Current Snapshot: MvccSnapshot[applied={T|T < 7333977920449634304}]
I20260927 15:12:33.284024 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5372ce963d494f678e6d1fa1804e4f3c. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.284408 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.289392 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.289726 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:33.290151 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.290377 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:33.290966 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.291275 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.293244 23458 catalog_manager.cc:1347] Generated new cluster ID: b508b6b21710451ea16e67a5026b130b
I20260927 15:12:33.293366 23458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:33.298259 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5372ce963d494f678e6d1fa1804e4f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5372ce963d494f678e6d1fa1804e4f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45889 } } peers { permanent_uuid: "515c6c7460fc4db7ae408f7c64d0bcde" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37319 } } peers { permanent_uuid: "f24bc51f6fdb487498d541f4a7bd1f5f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34501 } } }
I20260927 15:12:33.298550 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:12:33.337427 23458 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:33.339558 23458 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:33.362001 23458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5372ce963d494f678e6d1fa1804e4f3c: Generated new TSK 0
I20260927 15:12:33.363821 23458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:12:33.398584 23481 catalog_manager.cc:1259] Loaded cluster ID: b508b6b21710451ea16e67a5026b130b
I20260927 15:12:33.398830 23481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: loading cluster ID for follower catalog manager: success
I20260927 15:12:33.402999 23481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: acquiring CA information for follower catalog manager: success
I20260927 15:12:33.406474 23481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 515c6c7460fc4db7ae408f7c64d0bcde: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:12:33.503908 23420 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.479471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.97.62:45889)
0927 15:12:33.479702 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.479711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.501361 (+ 21650us) server_negotiation.cc:407] Connection header received
0927 15:12:33.502139 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.502146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.502243 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.502340 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.503610 (+ 1270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34224: BlockingRecv error: recv got EOF from 127.0.0.1:34224 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 15:12:33.507511 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.485809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.0.97.61:37319)
0927 15:12:33.486041 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.486052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.505100 (+ 19048us) server_negotiation.cc:407] Connection header received
0927 15:12:33.505683 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.505690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.505784 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.505865 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.507237 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52522: BlockingRecv error: recv got EOF from 127.0.0.1:52522 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:12:33.510396 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.500590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.0.97.60:34501)
0927 15:12:33.500838 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.500846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.508263 (+ 7417us) server_negotiation.cc:407] Connection header received
0927 15:12:33.508726 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.508732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.508822 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.508905 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.510105 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53026: BlockingRecv error: recv got EOF from 127.0.0.1:53026 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:12:33.564141 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.549638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.0.97.61:37319)
0927 15:12:33.549833 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.549840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.555767 (+ 5927us) server_negotiation.cc:407] Connection header received
0927 15:12:33.556197 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.556204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.556458 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.556618 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:33.557396 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.558569 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.560673 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.561525 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.561637 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.562455 (+ 818us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:33.563223 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:33.563371 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.563641 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.563761 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 15:12:33.626081 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.607517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.0.97.60:34501)
0927 15:12:33.607724 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.607733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.623825 (+ 16092us) server_negotiation.cc:407] Connection header received
0927 15:12:33.624222 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.624234 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.624394 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.624505 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.625812 (+ 1307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53032: BlockingRecv error: recv got EOF from 127.0.0.1:53032 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:12:33.679777 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.664463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.97.61:37319)
0927 15:12:33.664730 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.664737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.672334 (+ 7597us) server_negotiation.cc:407] Connection header received
0927 15:12:33.672910 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.672917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.673028 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.673100 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:33.673851 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.674988 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.677213 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.677967 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.678083 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.678803 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:33.678973 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:33.679047 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.679347 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.679451 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 15:12:33.737845 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.720676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.0.97.60:34501)
0927 15:12:33.720933 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.720940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.735797 (+ 14857us) server_negotiation.cc:407] Connection header received
0927 15:12:33.736012 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.736018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.736149 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.736283 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.737582 (+ 1299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53036: BlockingRecv error: recv got EOF from 127.0.0.1:53036 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:12:33.789500 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.774053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.0.97.61:37319)
0927 15:12:33.774287 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.774294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.781872 (+ 7578us) server_negotiation.cc:407] Connection header received
0927 15:12:33.782494 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.782500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.782610 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.782692 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:33.783572 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.784805 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.786968 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.787724 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.787838 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.788506 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:33.788662 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:33.788739 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.789001 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.789105 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 15:12:33.829883 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.821451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.0.97.60:34501)
0927 15:12:33.821666 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.821675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.827771 (+ 6096us) server_negotiation.cc:407] Connection header received
0927 15:12:33.828272 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.828280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.828405 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.828493 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.829603 (+ 1110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53050: BlockingRecv error: recv got EOF from 127.0.0.1:53050 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:12:33.880915 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.873803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.97.61:37319)
0927 15:12:33.874005 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.874011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.874604 (+ 593us) server_negotiation.cc:407] Connection header received
0927 15:12:33.874812 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.874819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.874925 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.875043 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:33.875535 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.876669 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.878683 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.879431 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.879575 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.880055 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:33.880201 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:33.880312 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.880469 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.880572 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:12:33.893445 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.888523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.97.60:34501)
0927 15:12:33.888908 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.888915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.891820 (+ 2905us) server_negotiation.cc:407] Connection header received
0927 15:12:33.892055 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.892061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.892163 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.892279 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.893208 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53058: BlockingRecv error: recv got EOF from 127.0.0.1:53058 (error 108)
Metrics: {"server-negotiator.queue_time_us":245}
I20260927 15:12:33.912501 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.906906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.97.61:37319)
0927 15:12:33.907150 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.907157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.907250 (+ 93us) server_negotiation.cc:407] Connection header received
0927 15:12:33.907543 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.907552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.907623 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.907731 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:33.908156 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.908952 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.910454 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.911027 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.911235 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.911669 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:33.911819 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:33.911918 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.912077 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.912161 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:12:33.929757 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.927402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.0.97.60:34501)
0927 15:12:33.927634 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.927641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.928017 (+ 376us) server_negotiation.cc:407] Connection header received
0927 15:12:33.928208 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.928212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.928375 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.928464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.929481 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53074: BlockingRecv error: recv got EOF from 127.0.0.1:53074 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:12:33.955497 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:33.949464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.97.61:37319)
0927 15:12:33.949745 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.949755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.950319 (+ 564us) server_negotiation.cc:407] Connection header received
0927 15:12:33.950563 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.950567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.950640 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.950742 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:33.951299 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.952266 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.953723 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:33.954165 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:33.954319 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:33.954746 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:33.954857 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:33.954954 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:33.955121 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 15:12:33.955218 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 15:12:33.967182 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:33.965061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.0.97.60:34501)
0927 15:12:33.965296 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:33.965303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:33.965628 (+ 325us) server_negotiation.cc:407] Connection header received
0927 15:12:33.965831 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:33.965836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:33.965911 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:33.966024 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:33.966896 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53088: BlockingRecv error: recv got EOF from 127.0.0.1:53088 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:12:34.009168 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.002683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.0.97.61:37319)
0927 15:12:34.002918 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.002928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.003286 (+ 358us) server_negotiation.cc:407] Connection header received
0927 15:12:34.003455 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.003460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.003570 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.003657 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.004128 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.005125 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.007363 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.007810 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.007924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.008422 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.008541 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.008610 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.008802 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.008872 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 15:12:34.017287 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:34.015276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.0.97.60:34501)
0927 15:12:34.015531 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.015541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.015919 (+ 378us) server_negotiation.cc:407] Connection header received
0927 15:12:34.016054 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.016059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.016134 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.016227 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:34.017019 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53090: BlockingRecv error: recv got EOF from 127.0.0.1:53090 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:12:34.089913 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.084773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.0.97.61:37319)
0927 15:12:34.085043 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.085049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.085253 (+ 204us) server_negotiation.cc:407] Connection header received
0927 15:12:34.085476 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.085481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.085553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.085638 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.086082 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.086830 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.088237 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.088656 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.088759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.089179 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.089310 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.089377 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.089508 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.089576 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260927 15:12:34.098392 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:12:34.096284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.0.97.60:34501)
0927 15:12:34.096576 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.096583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.096827 (+ 244us) server_negotiation.cc:407] Connection header received
0927 15:12:34.096985 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.096989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.097066 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.097155 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:34.098150 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53092: BlockingRecv error: recv got EOF from 127.0.0.1:53092 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 15:12:34.112169 23436 catalog_manager.cc:1259] Loaded cluster ID: b508b6b21710451ea16e67a5026b130b
I20260927 15:12:34.112269 23436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: loading cluster ID for follower catalog manager: success
I20260927 15:12:34.115124 23436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: acquiring CA information for follower catalog manager: success
I20260927 15:12:34.117144 23436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f24bc51f6fdb487498d541f4a7bd1f5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:12:34.236044 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.230122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.0.97.61:37319)
0927 15:12:34.230381 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.230390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.230736 (+ 346us) server_negotiation.cc:407] Connection header received
0927 15:12:34.230989 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.230993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.231086 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.231225 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.231895 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.232709 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.234222 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.234700 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.234844 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.235286 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.235402 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.235504 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.235656 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.235745 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:12:34.248713 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.242724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.0.97.60:34501)
0927 15:12:34.242964 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.242971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.243330 (+ 359us) server_negotiation.cc:407] Connection header received
0927 15:12:34.243586 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.243593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.243850 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.243990 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.244473 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.245199 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.246568 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.246967 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.247060 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.247636 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.248175 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.248301 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.248410 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.248479 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:12:34.266903 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.255673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.0.97.62:45889)
0927 15:12:34.258375 (+ 2702us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.258386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.258441 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:12:34.258656 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.258689 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.259125 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.260328 (+ 1203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.260606 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.261961 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.264308 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.264945 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.265114 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.265739 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.266204 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.266319 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.266465 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.266561 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2425,"thread_start_us":3957,"threads_started":1}
I20260927 15:12:34.267287 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.255837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.97.61:37319)
0927 15:12:34.256104 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.256114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.260513 (+ 4399us) server_negotiation.cc:407] Connection header received
0927 15:12:34.260747 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.260752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.260847 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.260952 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.261749 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.262819 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.265103 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.265729 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.265871 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.266294 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.266433 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.266510 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.266878 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.266959 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:12:34.269728 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:34.256970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.0.97.60:34501)
0927 15:12:34.257359 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:34.257368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:34.262754 (+ 5386us) server_negotiation.cc:407] Connection header received
0927 15:12:34.263115 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:34.263122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:34.263210 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:34.263319 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:12:34.263952 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.265002 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.266944 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:34.267658 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:34.267771 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:34.268910 (+ 1139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:12:34.269069 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:12:34.269163 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:34.269360 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 15:12:34.269426 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:12:34.272682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 23200
I20260927 15:12:34.273778 23262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:34.386377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 23200
W20260927 15:12:34.407945 23292 connection.cc:535] server connection from 127.0.0.1:52496 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:12:34.408475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 23274
I20260927 15:12:34.409741 23336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:34.521389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 23274
I20260927 15:12:34.538357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0r1zNn/build/asan/bin/kudu pid 23349
I20260927 15:12:34.539685 23411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:34.651063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0r1zNn/build/asan/bin/kudu with pid 23349
2026-09-27T15:12:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4648 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9339 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (377094 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST